Target object perception analysis method and system based on multi-modal data, medium and vehicle

By collecting and fusing multi-source heterogeneous data to generate target object perception features and constructing a dynamic knowledge graph, the problem of non-real-time acquisition of user survey data in existing technologies is solved, and the effect of rapid feedback on user needs is achieved.

CN121479573APending Publication Date: 2026-02-06CHINA FAW CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511788983.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-11-30
Publication Date
2026-02-06

AI Technical Summary

Technical Problem

Existing user research methods are unable to quickly obtain real-time and sufficient user perception data, resulting in the inability to make timely adjustments in the production process.

Method used

By collecting multi-source heterogeneous data (audio, images, text), classifying and fusing it, generating perceptual features of the target object, constructing a dynamic knowledge graph, and determining the analysis results of the target object.

Benefits of technology

It enables rapid acquisition of recent user perceptions, allowing for quick feedback of user needs and preferences to the production stage, and supporting timely adjustments in the production process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121479573A_ABST
    Figure CN121479573A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data processing, in particular to a target object perception analysis method and system based on multi-modal data, a medium and a vehicle. The method comprises the following steps: collecting multi-source heterogeneous data, wherein the multi-source heterogeneous data comprises audio data, image data and text data; classifying the multi-source heterogeneous data to obtain a plurality of multi-modal data packets; performing data fusion on the multi-modal data packet to obtain a target object sensing feature; generating scene description words, perception description words and emotion description words for the target object perception features according to a pre-trained large language model; constructing / updating a dynamic knowledge graph through triple data formed by the scene description words, the perception description words and the emotion description words; and determining a target object analysis result according to the dynamic knowledge graph. The target object analysis result obtained through the method can reflect recent user perception conditions, and user demands and preferences are fed back to a production section in the fastest time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a target object perception and analysis method, system, medium, and vehicle based on multimodal data. Background Technology

[0002] Conducting user research on marketed models can yield a wealth of user perception data, such as user needs and preferences. This data can be used to guide vehicle design optimization, market strategy development, and production process improvement.

[0003] Traditional user research methods involve distributing questionnaires to users and then analyzing the responses. However, with constantly evolving user needs and intensifying market competition, traditional questionnaires are no longer sufficient to obtain real-time and adequate sample sizes. Therefore, rapidly acquiring user perception data and making timely adjustments in the production process has become a pressing technical challenge. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a target object perception analysis method, system, medium, and vehicle based on multimodal data, which can quickly acquire user perception data through multi-source heterogeneous data on the network.

[0005] The first aspect of this application provides a target object perception and analysis method based on multimodal data fusion, including: Collect multi-source heterogeneous data, including audio data, image data, and text data; The multi-source heterogeneous data is classified to obtain several multimodal data packets; Data fusion is performed on the multimodal data packets to obtain the target object perception features; Based on the pre-trained large language model, scene description words, perception description words, and emotion description words are generated from the perceptual features of the target object. A dynamic knowledge graph is constructed / updated using triplet data composed of the scene descriptors, the perception descriptors, and the emotion descriptors. The analysis results of the target object are determined based on the dynamic knowledge graph.

[0006] Optionally, the collection of multi-source heterogeneous data specifically includes: Comments were crawled from the web as the text data. The driving image is extracted from the image and / or video uploaded by the target object as the image data; The audio track is extracted from the video uploaded by the target object as the audio data.

[0007] Optionally, classifying the multi-source heterogeneous data to obtain several multimodal data packets specifically includes: The multi-source heterogeneous data are classified according to the target object ID and vehicle model; The multi-source heterogeneous data is divided into the same multimodal data packet based on the same target object ID and the same vehicle model.

[0008] Optionally, data fusion is performed on the multimodal data packets to obtain target object perception features, specifically including: The multimodal data packets are preprocessed with text, images, and audio. The preprocessed text data, image data, and audio data are fused using a multimodal fusion neural network model to generate the target object perception features.

[0009] Optionally, the multimodal data packets undergo text preprocessing, image preprocessing, and audio preprocessing, specifically including: For the text data, long sentences are segmented into short sentences using a semantic segmentation algorithm; And / or, for the image data, image tags are identified using a target detection algorithm, the image tags including driving scene type and vehicle component type; And / or, for the audio data, it is converted into text with paralinguistic features using a speech recognition algorithm.

[0010] Optionally, the step of fusing the preprocessed text data, image data, and audio data using a multimodal fusion neural network model specifically includes: The short sentence text, the image label, and the text with paralinguistic features are input into the multimodal fusion neural network model for fusion.

[0011] Optionally, the analysis results of the target object are determined based on the dynamic knowledge graph, specifically including: By mapping the perceptual descriptive words whose frequency is greater than or equal to a preset frequency threshold using a preset rule base, the analysis results of the target object corresponding to the preset rule base are obtained.

[0012] A second aspect of this application provides a target object perception and analysis system based on multimodal data fusion, comprising: The data acquisition module is used to acquire multi-source heterogeneous data and classify the multi-source heterogeneous data to obtain several multimodal data packets; The data preprocessing and fusion module is used to perform data fusion on the multimodal data packets to obtain the target object perception features; The intelligent analysis module is used to generate scene description words, perception description words, and sentiment description words based on the perceptual features of the target object using a pre-trained large language model, and to construct / update a dynamic knowledge graph using the triplet data composed of the scene description words, the perceptual description words, and the sentiment description words. It is recommended to generate a module for determining the analysis results of the target object based on the dynamic knowledge graph.

[0013] A third aspect of this application provides a computer-readable storage medium storing computer-executable instructions for enabling the target object perception and analysis method described in the second aspect of this application.

[0014] The fourth aspect of this application provides a vehicle equipped with the computer-readable storage medium described in the third aspect of this application.

[0015] Beneficial effects: The method described in this application crawls heterogeneous audio, video, and text sources from the internet. After data classification, it fuses the data from these three sources to generate target object perception features. Extracting these target object perception features yields three descriptive terms for constructing triplet data. Constructing / updating a dynamic knowledge graph using triplet data allows for updating recent user perceptions. The target analysis results determined based on the updated dynamic knowledge graph accurately reflect recent user perception. This method enables rapid feedback of user needs and preferences to the production stage. Attached Figure Description

[0016] Figure 1 This is a flowchart of the target object perception and analysis method shown in the embodiments of this application; Figure 2 This is a flowchart of sub-step 101 of the target object perception and analysis method shown in the embodiments of this application; Figure 3 This is a flowchart of sub-step 102 of the target object perception and analysis method shown in the embodiments of this application; Figure 4 This is a flowchart of sub-step 103 of the target object perception and analysis method shown in the embodiments of this application; Figure 5 This is a structural diagram of the target object perception and analysis system shown in the embodiments of this application. Detailed Implementation

[0017] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0018] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the drawings and are only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0019] In the description of this invention, "multiple" refers to two or more. The use of "first" and "second" is for distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features or their sequential relationship.

[0020] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0021] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are some embodiments of the present invention, not all embodiments.

[0022] Traditional user research methods involve distributing questionnaires to users and then analyzing the responses. However, with constantly evolving user needs and intensifying market competition, traditional questionnaires are no longer sufficient to obtain real-time and adequate sample sizes. Therefore, rapidly acquiring user perception data and making timely adjustments in the production process has become a pressing technical challenge.

[0023] In view of this, this application provides a target object perception and analysis method, system, medium, and vehicle based on multimodal data, which will be specifically described through the following embodiments. First, the target object perception and analysis method in the embodiments of this application is described.

[0024] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0025] Figure 1This is a flowchart of the target object perception and analysis method described in the embodiments of this application.

[0026] Figure 1 The method shown includes, but is not limited to, steps 101 to 106.

[0027] 101. Collect multi-source heterogeneous data, including audio data, image data, and text data; 102. Classify the multi-source heterogeneous data to obtain several multimodal data packets; 103. Perform data fusion on the multimodal data packets to obtain the target object perception features; 104. Generate scene description words, perception description words, and sentiment description words based on the perceptual features of the target object using a pre-trained large language model; 105. Construct / update a dynamic knowledge graph using the triplet data composed of the scene description words, the perception description words, and the emotion description words; 106. Determine the analysis results of the target object based on the dynamic knowledge graph.

[0028] The method described in this application involves crawling heterogeneous audio, video, and text sources from the internet. After data classification, the data from these three sources is fused to generate target object perception features. Extracting these target object perception features yields three descriptive terms for constructing triplet data. Constructing / updating a dynamic knowledge graph using triplet data allows for updating recent user perceptions. The target analysis results determined based on the updated dynamic knowledge graph accurately reflect recent user perception. This method enables rapid feedback of user needs and preferences to the production stage.

[0029] In the triplet data of step 105, the triplet data is stored in a graph database according to "scene-perception-emotion". The scene descriptors and the perception descriptors are nodes, and the emotion descriptors are edges. The edge weight represents the co-occurrence frequency of the scene descriptors and the perception descriptors. For example, the scene descriptors can be "city road - congested following", "highway - emergency lane change", "mountain road - sharp bend", the perception descriptors can be "large steering wheel play", "sluggish throttle response", "insufficient seat support", and the emotion descriptors can be positive, negative, or neutral.

[0030] Once constructed, the dynamic knowledge graph has an automatic update function. By statistically analyzing the changes in the number of triples, frequency, and edge weights, it can discover potential, undefined associations (for example, it can be found that in the "rainy and snowy weather" scenario, users' perception of "ESP intervention" increases significantly).

[0031] In some embodiments, such as Figure 2As shown, step 101 may include, but is not limited to, steps 201 to 203. 201. Crawl comments from the internet as the text data; 202. Extract driving images from images and / or videos uploaded by the target object as the image data; 203. Extract the audio track from the video uploaded by the target object as the audio data.

[0032] In steps 201-203, text data can be obtained by web crawling from car forums, social media, and video platforms. Image data such as vehicle interior, driving environment, and dashboard can be extracted from user-uploaded pictures and short videos. Audio tracks can be extracted from videos, or audio data can be directly collected from voice feedback platforms.

[0033] In some embodiments, such as Figure 3 As shown, step 102 may include, but is not limited to, steps 301 to 302.

[0034] 301. Classify several of the aforementioned multi-source heterogeneous data based on the target object ID and vehicle model; 302. Divide the multi-source heterogeneous data into the same multimodal data packet based on the same target object ID and the same vehicle model.

[0035] In some embodiments, such as Figure 4 As shown, step 103 may include, but is not limited to, steps 401 to 402.

[0036] 401. Perform text preprocessing, image preprocessing, and audio preprocessing on the multimodal data packets; 402. The preprocessed text data, image data, and audio data are fused using a multimodal fusion neural network model to generate the target object perception features.

[0037] In step 401, for the text data, long sentences are segmented into short sentences using a semantic segmentation algorithm. For the image data, image labels are identified using an object detection algorithm; these labels include driving scene type and vehicle component type. For the audio data, it is converted into text with paralinguistic features using a speech recognition algorithm.

[0038] For example, the object detection algorithm can be the YOLO algorithm, and the speech recognition algorithm can be an open-source deep learning network such as Whisper.

[0039] Step 402 specifically includes: inputting the short sentence text, the image label, and the text with paralinguistic features into the multimodal fusion neural network model for fusion.

[0040] In this embodiment, the data extracted from the above multiple modalities are fused using a Transformer-based architecture to generate a unified feature representation rich in contextual information.

[0041] In some embodiments, step 106 specifically includes: performing rule mapping on the perceptual descriptors whose frequency of occurrence is greater than or equal to a preset frequency threshold through a preset rule base, and obtaining the target object analysis result corresponding to the preset rule base.

[0042] The pre-defined rule base contains mapping rules between "perceived problems and design directions" (for example, if "fuzzy steering feel" occurs frequently, it is recommended to "check the steering gear ratio or electronic power steering logic"). Combining the analysis results from the knowledge graph, the system automatically matches and generates a preliminary design optimization suggestion report.

[0043] This application also provides a target object perception and analysis system based on multimodal data, used to execute the target object perception and analysis method described in the above embodiments, such as... Figure 5 As shown, it includes: Data acquisition module 1001 is used to acquire multi-source heterogeneous data and classify the multi-source heterogeneous data to obtain several multimodal data packets; The data preprocessing and fusion module 1002 is used to perform data fusion on the multimodal data packets to obtain the target object perception features; The intelligent analysis module 1003 is used to generate scene description words, perception description words, and sentiment description words based on the perceptual features of the target object according to the pre-trained large language model, and to construct / update a dynamic knowledge graph using the triplet data composed of the scene description words, the perceptual description words, and the sentiment description words. It is recommended to generate module 1004, which is used to determine the analysis results of the target object based on the dynamic knowledge graph.

[0044] This application also provides a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the target object perception and analysis method described in the above embodiments.

[0045] This application also provides a vehicle equipped with the data collection system described in the above embodiments. The vehicle can be a private car, such as a sedan, SUV, MPV, or pickup truck. It can also be a commercial vehicle, such as a van, bus, small truck, or large trailer. The vehicle needs to have an electric motor capable of outputting power or acting as a generator to store mechanical energy. When the vehicle is a new energy vehicle, it can be a hybrid vehicle or a pure electric vehicle.

[0046] Since the vehicle applies all the technical solutions of the above-described target object perception and analysis method or target object perception and analysis system, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be repeated here.

[0047] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0048] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A target object perception and analysis method based on multimodal data, characterized in that, include: Collect multi-source heterogeneous data, including audio data, image data, and text data; The multi-source heterogeneous data is classified to obtain several multimodal data packets; Data fusion is performed on the multimodal data packets to obtain the target object perception features; Based on the pre-trained large language model, scene description words, perception description words, and emotion description words are generated from the perceptual features of the target object. A dynamic knowledge graph is constructed / updated using triplet data composed of the scene descriptors, the perception descriptors, and the emotion descriptors. The analysis results of the target object are determined based on the dynamic knowledge graph.

2. The target object perception and analysis method as described in claim 1, characterized in that, The collection of multi-source heterogeneous data specifically includes: Comments were crawled from the web as the text data. The driving image is extracted from the image and / or video uploaded by the target object as the image data; The audio track is extracted from the video uploaded by the target object as the audio data.

3. The target object perception and analysis method as described in claim 1, characterized in that, The classification of the multi-source heterogeneous data to obtain several multimodal data packets specifically includes: The multi-source heterogeneous data are classified according to the target object ID and vehicle model; The multi-source heterogeneous data is divided into the same multimodal data packet based on the same target object ID and the same vehicle model.

4. The target object perception and analysis method as described in claim 1, characterized in that, Data fusion is performed on the multimodal data packets to obtain target object perception features, specifically including: The multimodal data packets are preprocessed with text, images, and audio. The preprocessed text data, image data, and audio data are fused using a multimodal fusion neural network model to generate the target object perception features.

5. The target object perception and analysis method as described in claim 4, characterized in that, The multimodal data packets undergo text preprocessing, image preprocessing, and audio preprocessing, specifically including: For the text data, long sentences are segmented into short sentences using a semantic segmentation algorithm; And / or, for the image data, image tags are identified using a target detection algorithm, the image tags including driving scene type and vehicle component type; And / or, for the audio data, it is converted into text with paralinguistic features using a speech recognition algorithm.

6. The target object perception and analysis method as described in claim 5, characterized in that, The process of fusing the preprocessed text data, image data, and audio data using a multimodal fusion neural network model specifically includes: The short sentence text, the image label, and the text with paralinguistic features are input into the multimodal fusion neural network model for fusion.

7. The target object perception and analysis method as described in claim 1, characterized in that, The analysis results of the target object are determined based on the dynamic knowledge graph, specifically including: By mapping the perceptual descriptive words whose frequency is greater than or equal to a preset frequency threshold using a preset rule base, the analysis results of the target object corresponding to the preset rule base are obtained.

8. A target object perception and analysis system based on multimodal data, characterized in that, include: The data acquisition module is used to acquire multi-source heterogeneous data and classify the multi-source heterogeneous data to obtain several multimodal data packets; The data preprocessing and fusion module is used to perform data fusion on the multimodal data packets to obtain the target object perception features; The intelligent analysis module is used to generate scene description words, perception description words, and sentiment description words based on the perceptual features of the target object using a pre-trained large language model, and to construct / update a dynamic knowledge graph using the triplet data composed of the scene description words, the perceptual description words, and the sentiment description words. It is recommended to generate a module for determining the analysis results of the target object based on the dynamic knowledge graph.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the target object perception and analysis method as described in any one of claims 1 to 7.

10. A vehicle, characterized in that, The computer-readable storage medium of claim 9 is provided.