A data integration method and related devices

Abstract

This application discloses a data integration method and related devices. During the driving process of a vehicle, semantic analysis can be respectively performed on the acquired real-time vehicle data of different modalities to obtain the semantic representation results of the real-time vehicle data of different modalities, and the semantic representation results of the real-time vehicle data of different modalities are input into a classification model for verifying whether the semantic representation results of the real-time vehicle data of different modalities match each other. When the result output by the classification model indicates that the semantic representation results of the real-time vehicle data of different modalities match, it can be determined that the data integration of the real-time vehicle data of different modalities has been completed, and the semantic alignment of the real-time vehicle data of different modalities is achieved. In this way, the real-time vehicle data of different modalities can be integrated to improve the accuracy of subsequent vehicle data analysis. Further, by accessing the first weight subgraph, the integration of continuous key frame data is realized, and the processing ability of information within a continuous time is enhanced.

Description

Technical Field

[0001] This application relates to the technical field of vehicle control, and particularly relates to a data integration method and related devices. Background Art

[0002] In recent years, with the development of emerging technologies such as big data and cloud computing, the forms of data existence have gradually shown a trend of diversification, and different modal data such as text, pictures, and videos have grown rapidly. Although these data are in different modalities, they may be highly relevant in specific tasks or applications and can be used to represent the same semantics.

[0003] Taking vehicle data as an example, its data types are complex and diverse, including map data, environmental data, battery data, etc. These data are mainly composed of numerical modality, text modality, and image modality, etc., and can describe the vehicle running state at the same moment from different perspectives. However, since it is difficult to link different modal vehicle data, in the process of vehicle data analysis, generally, the vehicle data within a single modality is analyzed separately to obtain the vehicle running state represented by the vehicle data within that modality, and it is impossible to analyze using different modal data simultaneously, resulting in insufficient accuracy of vehicle data analysis. Summary of the Invention

[0004] Embodiments of this application provide a data integration method and related devices, which integrate real-time vehicle data of different modalities, solve the problem of insufficient accuracy of vehicle data analysis caused by the inability to analyze using different modal data simultaneously, and can enhance the ability to extract and process information within continuous time.

[0005] In a first aspect, embodiments of this application provide a data integration method, including:

[0006] During vehicle driving, obtain real-time vehicle data of different modalities;

[0007] Perform semantic analysis on the real-time vehicle data of different modalities respectively to obtain semantic representation results of the real-time vehicle data of different modalities;

[0008] Input the semantic representation results of the real-time vehicle data of different modalities into a classification model; the classification model is used to verify whether the semantic representation results of the real-time vehicle data of different modalities match each other;

[0009] When the result output by the classification model indicates that the semantic representation results of the real-time vehicle data of different modalities match each other, determine that the data integration of the real-time vehicle data of different modalities has been completed.

[0010] Optionally, the real-time vehicle data of different modalities includes real-time vehicle data of image modality related to the vehicle exterior environment;

[0011] Semantically analyzing the real-time vehicle data of the different modalities respectively to obtain the semantic representation results of the real-time vehicle data of the different modalities, including:

[0012] Extracting a plurality of consecutive key frame data from the real-time vehicle data of the image modality, and performing image recognition on the plurality of consecutive key frame data to obtain multiple groups of recognition results related to the external environment of the vehicle;

[0013] Analyzing the multiple groups of recognition results respectively to obtain the semantic representation results of the real-time vehicle data of the image modality.

[0014] Optionally, the analyzing the multiple groups of recognition results respectively to obtain the semantic representation results of the real-time vehicle data of the image modality includes:

[0015] Obtaining the information quantity weights of the plurality of consecutive key frame data respectively according to the multiple groups of recognition results;

[0016] Constructing a first weight subgraph corresponding to the information quantity weights of the plurality of consecutive key frame data according to the multiple groups of recognition results and the information quantity weights of the plurality of consecutive key frame data; wherein, multiple nodes in the first weight subgraph respectively include the multiple groups of recognition results; the plurality of consecutive key frame data respectively carry time information;

[0017] Traversing and accessing the first weight subgraph, and determining the masking probability of the multiple groups of recognition results changing with time according to the information quantity weights of the plurality of consecutive key frame data; the masking probability is used to describe the probability situation of the first weight subgraph being masked when accessing from one node in the first weight subgraph to the next node;

[0018] Updating the first weight subgraph by using the masking probability, and taking the updated first weight subgraph as the semantic representation results of the real-time vehicle data of the image modality.

[0019] Optionally, the obtaining the information quantity weights of the plurality of consecutive key frame data according to the multiple groups of recognition results includes:

[0020] Obtaining the prior probabilities, confidence levels and recognition frame sizes respectively corresponding to the multiple groups of recognition results;

[0021] Determining the information quantity weights of the plurality of consecutive key frame data respectively according to the prior probabilities, confidence levels and recognition frame sizes respectively corresponding to the multiple groups of recognition results.

[0022] Optionally, the real-time vehicle data of the different modalities includes real-time vehicle data of the numerical modality;

[0023] Performing semantic analysis on the real-time vehicle data of the different modalities respectively to obtain the semantic representation results of the real-time vehicle data of the different modalities, including:

[0024] Cutting the real-time vehicle data of the numerical modality at a preset time interval to obtain at least two cutting results;

[0025] Encoding the at least two cutting results respectively, and using the obtained encoding results as the semantic representation results of the real-time vehicle data of the numerical modality.

[0026] Optionally, the above data integration method further includes:

[0027] Predicting the vehicle working conditions based on the real-time vehicle data of different modalities after data integration.

[0028] Optionally, the above data integration method further includes:

[0029] Identifying the vehicle working conditions based on the real-time vehicle data of different modalities after data integration to obtain the working condition identification results corresponding to the real-time vehicle data of different modalities after data integration respectively;

[0030] Calibrating the real-time vehicle data of different modalities after data integration respectively according to the working condition identification results corresponding to the real-time vehicle data of different modalities after data integration and based on the preset vehicle working condition labels associated with the vehicle data.

[0031] In a second aspect, an embodiment of the present application provides a data integration device, including:

[0032] A vehicle data acquisition module, configured to acquire real-time vehicle data of different modalities during vehicle driving;

[0033] A semantic representation result acquisition module, configured to perform semantic analysis on the real-time vehicle data of the different modalities respectively to obtain the semantic representation results of the real-time vehicle data of the different modalities;

[0034] A classification module, configured to input the semantic representation results of the real-time vehicle data of the different modalities into a classification model; the classification model is used to verify whether the semantic representation results of the real-time vehicle data of the different modalities match each other;

[0035] A data integration determination module, configured to determine that the data integration of the real-time vehicle data of the different modalities has been completed when the result output by the classification model indicates that the semantic representation results of the real-time vehicle data of the different modalities match.

[0036] In a third aspect, an embodiment of the present application provides a data integration device, including a processor and a memory; the memory is used to store a computer program; the processor is used to execute the data integration method described above according to the computer program.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, and when the computer program is run by a processor, it executes the above data integration method.

[0038] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0039] In the embodiments of the present application, during the driving process of the vehicle, semantic analysis can be respectively performed on the acquired real-time vehicle data of different modalities to obtain the semantic representation results of the real-time vehicle data of different modalities, and the semantic representation results of the real-time vehicle data of different modalities are input into a classification model for verifying whether the semantic representation results of the real-time vehicle data of different modalities match each other. When the result output by the classification model indicates that the semantic representation results of the real-time vehicle data of different modalities match, it can be determined that the data integration of the real-time vehicle data of different modalities has been completed, and the semantic alignment of the real-time vehicle data of different modalities is realized. In this way, through semantic analysis of the real-time vehicle data of different modalities, the real-time vehicle data of different modalities can be integrated, so as to realize the semantic alignment of different modality data, so as to utilize different modality data for analysis simultaneously in the future, thereby improving the accuracy of vehicle data analysis, and being able to enhance the ability to extract and process information within a continuous time period. Description of the Drawings

[0040] Figure 1 It is a flowchart of a data integration method provided by an embodiment of the present application;

[0041] Figure 2 It is a flowchart of another data integration method provided by an embodiment of the present application;

[0042] Figure 3 It is a schematic structural diagram of a data integration device provided by an embodiment of the present application. Detailed Embodiments

[0043] As described above, the inventors found in the research on vehicle data that: the types of vehicle data are complex and diverse, including map data, environmental data, battery data, etc. These data are mainly composed of numerical modalities, text modalities, image modalities, etc., and can describe the vehicle operating state at the same moment from different perspectives. However, since it is difficult to link vehicle data of different modalities, in the process of vehicle data analysis, generally, the vehicle data within a single modality is analyzed separately to obtain the vehicle operating state represented by the vehicle data within that modality, and it is impossible to analyze using data of different modalities simultaneously, resulting in insufficient accuracy of vehicle data analysis.

[0044] To solve the above problems, an embodiment of the present application provides a data integration method, which includes: during vehicle driving, semantic analysis can be respectively performed on the acquired real-time vehicle data of different modalities to obtain the semantic representation results of the real-time vehicle data of different modalities, and the semantic representation results of the real-time vehicle data of different modalities are input into a classification model for verifying whether the semantic representation results of the real-time vehicle data of different modalities match each other. When the result output by the classification model indicates that the semantic representation results of the real-time vehicle data of different modalities match, it can be determined that the data integration of the real-time vehicle data of different modalities has been completed, and the semantic alignment of the real-time vehicle data of different modalities is achieved.

[0045] In this way, by performing semantic analysis on the real-time vehicle data of different modalities, the real-time vehicle data of different modalities can be integrated, thereby achieving the semantic alignment of data of different modalities, so as to subsequently analyze using data of different modalities simultaneously, thereby improving the accuracy of vehicle data analysis. In addition, there may be various problems with the real-time vehicle data of different modalities, such as the image acquisition volume not being proportional to the amount of information, abnormal data existing in the in-vehicle battery management system, invalid scenarios existing in the map data, etc. By integrating the real-time vehicle data of different modalities, data of other modalities can be used to make up for the problematic data, thereby effectively suppressing data interference in the subsequent data analysis process, eliminating potential noise, and further improving the accuracy of vehicle data analysis.

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0047] Figure 1 It is a flowchart of a data integration method provided by an embodiment of the present application. Combined withFigure 1 As shown in Figure 1 , the data integration method provided by the embodiments of the present application may include:

[0048] S101: During the driving of the vehicle, obtain real-time vehicle data of different modalities.

[0049] Here, the real-time vehicle data of different modalities can specifically be embodied as real-time vehicle data in numerical modality and real-time vehicle data in image modality. For example, the environmental data obtained by in-vehicle sensors is real-time vehicle data in image modality; the battery data obtained by the in-vehicle battery management system, such as the current, voltage, temperature, etc. of the battery pack, and the vehicle driving data obtained by vehicle-end sensors, such as vehicle speed, heading, steering wheel angle, etc. are all real-time vehicle data in numerical modality.

[0050] In addition, the present application does not specifically limit the acquisition method of real-time vehicle data of different modalities. Taking the environmental data obtained by in-vehicle sensors as an example, the environmental data sent by the in-vehicle sensors can be directly received, or the environmental data can be first uploaded to the cloud server, and the environmental data can be obtained by the method of the cloud server sending information.

[0051] S102: Perform semantic analysis on the real-time vehicle data of different modalities respectively to obtain the semantic representation results of the real-time vehicle data of different modalities.

[0052] Here, the embodiments of the present application can illustrate the semantic analysis process according to different cases of real-time vehicle data of different modalities.

[0053] In one case, the real-time vehicle data of different modalities includes real-time vehicle data in image modality related to the vehicle exterior environment. Correspondingly, S102 may specifically include: extracting multiple consecutive key frame data from the real-time vehicle data in image modality, performing image recognition on the multiple consecutive key frame data to obtain multiple groups of recognition results related to the vehicle exterior environment; analyzing each group of recognition results respectively to obtain the semantic representation results of the real-time vehicle data in image modality. Since during the smooth driving of the vehicle, the similarity of the real-time vehicle data of consecutive frames is relatively high, and its contribution to data integration is relatively small, therefore, the real-time vehicle data in image modality related to the vehicle exterior environment can first select the real-time vehicle data in image modality with higher importance through the technology of extracting key frames, and perform subsequent semantic analysis operations on it, so as to reduce the amount of data that needs to be integrated and improve the data processing efficiency.

[0054] Performing image recognition on multiple consecutive key-frame data, the multiple sets of recognition results related to the external environment of the vehicle can specifically include the recognition results of objects in the external environment and the relative spatial positions between the objects. Additionally, for the analysis process of the multiple sets of recognition results, the embodiments of the present application may not make specific limitations. For ease of understanding, a possible implementation manner is described below.

[0055] As a possible implementation manner, analyze the multiple sets of recognition results respectively to obtain the semantic representation results of the real-time vehicle data in the image modality, which can specifically include: obtaining the information quantity weights of multiple consecutive key-frame data respectively according to the multiple sets of recognition results; constructing a first weight sub-graph corresponding to the information quantity weights of multiple consecutive key-frame data according to the multiple sets of recognition results and the information quantity weights of multiple consecutive key-frame data; wherein, multiple nodes in the first weight sub-graph respectively include multiple sets of recognition results; multiple consecutive key-frame data respectively carry time information; traversing and accessing the first weight sub-graph, and determining the masking probability of the multiple sets of recognition results changing with time according to the information quantity weights of multiple consecutive key-frame data; the masking probability is used to describe the probability situation of the first weight sub-graph being masked when accessing from one node in the first weight sub-graph to the next node; updating the first weight sub-graph by using the masking probability, and taking the updated first weight sub-graph as the semantic representation result of the real-time vehicle data in the image modality. In this way, when traversing and accessing the first weight sub-graph, the situation of the masking process of the multiple sets of recognition results changing with time can be obtained. Here, the masking process refers to the situation of covering part and / or all of the multiple sets of recognition results as time changes. For example, when traversing and accessing the first weight sub-graph, it can be obtained that the data in a certain node is "at 8 o'clock in the morning on the same day, object A is 500 meters directly in front of the vehicle", and the data in the next node adjacent to this node is "at 8:05 in the morning on the same day, object B is 500 meters directly in front of the vehicle". Correspondingly, the situation of the masking process changing with time can be reflected as object B covering object A. Further, according to the information quantity weights of multiple consecutive key-frames, the masking probability of the multiple sets of recognition results changing with time can be determined to update the first weight sub-graph and complete the integration of the real-time vehicle data in the image modality.

[0056] In addition, the process of obtaining the information quantity weights of multiple consecutive key-frame data can specifically include: obtaining the prior probabilities, confidence levels, and recognition frame sizes respectively corresponding to the multiple sets of recognition results; determining the information quantity weights of multiple consecutive key-frame data respectively according to the prior probabilities, confidence levels, and recognition frame sizes respectively corresponding to the multiple sets of recognition results. Here, for the specific implementation manner of determining the information quantity weights of multiple continuously associated frame data, the embodiments of the present application may not make specific limitations. For example, the information quantity weights of multiple consecutive key-frame data can be calculated by the following formula:

[0057]

[0058] Among them, w is the information quantity weight, i is the recognition result, p(i) is the prior probability corresponding to the recognition result, k is the number of nodes passed on the traversal path, and x k is the confidence level, s is the area of the recognition frame, and n_DFS is the total number of nodes passed on the traversal path.

[0059] In another case, the real-time vehicle data of different modalities includes the real-time vehicle data of the data modality. Accordingly, S102 may specifically include: cutting the real-time vehicle data of the numerical modality at a preset time interval to obtain at least two cutting results; respectively encoding the at least two cutting results, and using the respectively obtained encoding results as the semantic representation results of the real-time vehicle data of the numerical modality. Since the real-time vehicle data of the numerical modality can be collected by different sampling devices and the sampling accuracies are also different, its instability is relatively high. Therefore, the real-time sampled numerical modality vehicle data can be cut by the time slicing technology, and the cutting results can be encoded, so as to complete the integration of the real-time vehicle data of the numerical modality.

[0060] In addition, in order to improve the accuracy of the semantic representation results of the real-time vehicle data of the numerical modality, the embodiments of the present application may also analyze the respectively obtained encoding results, determine the data relationship between the respectively obtained encoding results, and update the encoding results using the data relationship. Among them, the data relationship between the respectively obtained encoding results may be reflected as the similarity between the respectively obtained encoding results, or may be an implicit relationship determined according to specific data analysis tasks. The embodiments of the present application do not make specific limitations on this. In addition, the analysis of the encoding results can be realized through a joint training model of the seq2seq model and clustering.

[0061] S103: Input the semantic representation results of the real-time vehicle data of different modalities into the classification model.

[0062] The classification model is used to verify whether the semantic representation results of the real-time vehicle data of different modalities match each other. Specifically, the classification model may be a pre-trained transformer model.

[0063] S104: When the result output by the classification model indicates that the semantic representation results of the real-time vehicle data of different modalities match each other, it is determined that the data integration of the real-time vehicle data of different modalities has been completed.

[0064] As can be seen from the relevant content of S101-S104 above, in the embodiments of the present application, during the vehicle driving process, semantic analysis can be respectively performed on the obtained vehicle real-time data of different modalities to obtain the semantic representation results of the vehicle real-time data of different modalities, and the semantic representation results of the vehicle real-time data of different modalities are input into a classification model for verifying whether the semantic representation results of the vehicle real-time data of different modalities match each other. When the result output by the classification model indicates that the semantic representation results of the vehicle real-time data of different modalities match, it can be determined that the data integration of the vehicle real-time data of different modalities has been completed, and the semantic alignment of the vehicle real-time data of different modalities is realized. In this way, through semantic analysis of the vehicle real-time data of different modalities, the vehicle real-time data of different modalities can be integrated, so as to realize the semantic alignment of different modality data, so as to utilize the data of different modalities for analysis simultaneously subsequently, thereby improving the accuracy of vehicle data analysis. In addition, there may be various problems in the vehicle real-time data of different modalities, such as the image acquisition volume not being proportional to the amount of information, abnormal data existing in the in-vehicle battery management system, and invalid scenarios existing in the map data. By integrating the vehicle real-time data of different modalities, the data with problems can be compensated by using the data of other modalities, so that the data interference in the subsequent data analysis process can be effectively suppressed, potential noise can be eliminated, and the accuracy of vehicle data analysis can be further improved.

[0065] It can be understood that, in order to improve the analysis accuracy of vehicle data, in the embodiments of the present application, the vehicle working conditions can be predicted according to the vehicle real-time data of different modalities after data integration. In this way, by using a variety of vehicle real-time data of different modalities after data integration for vehicle working condition prediction, the problem of inaccurate data analysis caused by only using the vehicle real-time data within a single modality for analysis can be avoided.

[0066] In order to expand the usability of the vehicle real-time data of different modalities and improve the user experience, the embodiments of the present application can also provide another data integration method. The following will respectively describe this data integration method in combination with the embodiments and the drawings.

[0067] Figure 2 It is a flowchart of another data integration method provided by the embodiments of the present application. In combination with Figure 2 as shown, the data integration method provided by the embodiments of the present application may include:

[0068] S201: During the vehicle driving process, obtain the vehicle real-time data of different modalities.

[0069] S202: Respectively perform semantic analysis on the vehicle real-time data of different modalities to obtain the semantic representation results of the vehicle real-time data of different modalities.

[0070] S203: Input the semantic representation results of real-time vehicle data in different modalities into the classification model.

[0071] S204: When the results output by the classification model indicate that the semantic representation results of real-time vehicle data in different modalities match each other, it is determined that the data integration of real-time vehicle data in different modalities has been completed.

[0072] S205: Identify the vehicle operating conditions based on the real-time vehicle data in different modalities after data integration, and obtain the operating condition identification results corresponding to the real-time vehicle data in different modalities after data integration.

[0073] Here, the vehicle operating conditions may include vehicle driving conditions, such as smooth driving, emergency braking, congestion, lane change, etc., or may include vehicle equipment conditions, such as abnormal changes in battery capacity, abnormal decline in state of charge of the battery, etc. In this regard, the embodiments of the present application do not make specific limitations.

[0074] S206: Based on the operating condition identification results corresponding to the real-time vehicle data in different modalities after data integration, and based on the pre-set vehicle operating condition tags associated with the vehicle data, calibrate the real-time vehicle data in different modalities after data integration respectively.

[0075] Here, using the pre-set vehicle operating condition tags to calibrate the real-time vehicle data in different modalities after data integration respectively is convenient for subsequent construction of a database of vehicle data in different modalities, so that users can directly obtain the calibrated operating conditions corresponding to the vehicle data from the database, improving the user experience and realizing the expansion of the usability of real-time vehicle data in different modalities.

[0076] Based on the data integration method provided in the foregoing embodiments, correspondingly, the embodiments of the present application also provide a data integration device. The data integration device will be described below in combination with the embodiments and the drawings respectively.

[0077] Figure 3 It is a schematic structural diagram of an acquisition device for a test script provided by an embodiment of the present application. In combination with Figure 3 As shown, the data integration device 300 provided by the embodiments of the present application may include:

[0078] A vehicle data acquisition module 301, configured to acquire real-time vehicle data in different modalities during vehicle driving;

[0079] A semantic representation result acquisition module 302, configured to perform semantic analysis on the real-time vehicle data in different modalities respectively to obtain the semantic representation results of the real-time vehicle data in different modalities;

[0080] A classification module 303, configured to input the semantic representation results of vehicle real-time data in different modalities into a classification model; the classification model is used to verify whether the semantic representation results of vehicle real-time data in different modalities match each other;

[0081] A data integration determination module 304, configured to determine that the data integration of vehicle real-time data in different modalities has been completed when the results output by the classification model indicate that the semantic representation results of vehicle real-time data in different modalities match.

[0082] Combined with the relevant content of the above data integration device 300, it can be seen that the embodiment of the present application provides a data integration device 300. By performing semantic analysis on vehicle real-time data in different modalities through the vehicle data acquisition module 301, the semantic representation result acquisition module 302, the classification module 303, and the data integration determination module 304, the vehicle real-time data in different modalities can be integrated, so as to achieve semantic alignment of different modality data, so as to utilize different modality data for analysis simultaneously subsequently, thereby improving the accuracy of vehicle data analysis.

[0083] As an implementation manner, in order to integrate vehicle data in different modalities, the vehicle real-time data in different modalities may include vehicle real-time data in an image modality related to the external environment of the vehicle. Correspondingly, the semantic representation result acquisition module 302 may specifically include:

[0084] An identification result acquisition module, configured to extract multiple consecutive key frame data from the vehicle real-time data in the image modality, and perform image identification on the multiple consecutive key frame data to obtain multiple groups of identification results related to the external environment of the vehicle;

[0085] An identification result analysis module, configured to analyze each of the multiple groups of identification results to obtain the semantic representation result of the vehicle real-time data in the image modality.

[0086] As an implementation manner, in order to integrate vehicle data in different modalities, the identification result analysis module may specifically include:

[0087] An information amount weight acquisition module, configured to respectively obtain the information amount weights of multiple consecutive key frame data according to the multiple groups of identification results;

[0088] A weight sub-graph construction module, configured to construct a first weight sub-graph corresponding to the information amount weights of multiple consecutive key frame data according to the multiple groups of identification results and the information amount weights of multiple consecutive key frame data; wherein, multiple nodes in the first weight sub-graph are respectively multiple groups of identification results; multiple consecutive key frame data respectively carry time information;

[0089] The weight sub - graph traversal module is used to traverse and access the first weight sub - graph to obtain the probability of the mask changing with time information; the probability of the mask is used to describe the probability situation of the first weight sub - graph being masked when accessing from one node in the first weight sub - graph to the next node.

[0090] The weight sub - graph update module is used to update the first weight sub - graph by using the probability of the mask, and use the updated first weight sub - graph as the semantic representation result of the vehicle real - time data in the image modality.

[0091] As an implementation, in order to integrate vehicle data of different modalities, the information - quantity weight acquisition module is specifically used for:

[0092] Obtain the prior probabilities, confidences, and recognition box sizes respectively corresponding to multiple groups of recognition results.

[0093] Determine the information - quantity weights of multiple consecutive key - frame data respectively according to the prior probabilities, confidences, and recognition box sizes respectively corresponding to multiple groups of recognition results.

[0094] As an implementation, in order to integrate vehicle data of different modalities, the vehicle real - time data of different modalities includes the vehicle real - time data in the numerical modality. Correspondingly, the semantic representation result acquisition module 302 may specifically include:

[0095] The cutting result acquisition module is used to cut the vehicle real - time data in the numerical modality at a preset time interval to obtain at least two cutting results.

[0096] The cutting result encoding module is used to encode at least two cutting results respectively, and use the respectively obtained encoding results as the semantic representation results of the vehicle real - time data in the numerical modality.

[0097] As an implementation, in order to integrate vehicle data of different modalities, the data integration device 300 may further include:

[0098] The vehicle working condition prediction module is used to predict the vehicle working condition according to the vehicle real - time data of different modalities after data integration.

[0099] As an implementation, in order to integrate vehicle data of different modalities, the data integration device 300 may further include:

[0100] The vehicle working condition recognition module is used to recognize the vehicle working condition according to the vehicle real - time data of different modalities after data integration, and obtain the working condition recognition results respectively corresponding to the vehicle real - time data of different modalities after data integration.

[0101] A vehicle real-time data calibration module is used to calibrate the vehicle real-time data of different modalities after data integration respectively according to the working condition recognition results corresponding to the vehicle real-time data of different modalities after data integration, and based on the preset vehicle working condition tags associated with the vehicle data.

[0102] Based on the data integration method and device provided in the foregoing embodiments, correspondingly, the present application further provides a data integration device, including a processor and a memory; the memory is used to store a computer program; the processor is used to execute the data integration method provided in the method embodiment as described above according to the computer program.

[0103] Based on the data integration method, device and equipment provided in the foregoing embodiments, correspondingly, the present application further provides a computer-readable storage medium for storing a computer program, and the computer program, when run by a processor, executes the data integration method provided in the method embodiment as described above.

[0104] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data integration method, characterized in that, including: During the vehicle driving process, obtaining real-time vehicle data in different modalities; Performing semantic analysis on the real-time vehicle data in different modalities respectively to obtain semantic representation results of the real-time vehicle data in different modalities; Inputting the semantic representation results of the real-time vehicle data in different modalities into a classification model; the classification model is used to verify whether the semantic representation results of the real-time vehicle data in different modalities match each other; When the result output by the classification model indicates that the semantic representation results of the real-time vehicle data in different modalities match each other, determining that data integration of the real-time vehicle data in different modalities has been completed, and the data integration of the real-time vehicle data in different modalities is used to represent semantic alignment of the real-time vehicle data in different modalities.

2. The method according to claim 1, wherein The real-time vehicle data in different modalities includes real-time vehicle data in an image modality related to the vehicle external environment; The performing semantic analysis on the real-time vehicle data in different modalities respectively to obtain semantic representation results of the real-time vehicle data in different modalities includes: Extracting multiple consecutive key frame data from the real-time vehicle data in the image modality and performing image recognition on the multiple consecutive key frame data to obtain multiple groups of recognition results related to the vehicle external environment; Analyzing the multiple groups of recognition results respectively to obtain the semantic representation result of the real-time vehicle data in the image modality.

3. The method according to claim 2, wherein The analyzing the multiple groups of recognition results respectively to obtain the semantic representation result of the real-time vehicle data in the image modality includes: Obtaining the information quantity weights of the multiple consecutive key frame data respectively according to the multiple groups of recognition results; Constructing a first weight subgraph corresponding to the information quantity weights of the multiple consecutive key frame data according to the multiple groups of recognition results and the information quantity weights of the multiple consecutive key frame data; wherein, multiple nodes in the first weight subgraph respectively include the multiple groups of recognition results; the multiple consecutive key frame data respectively carry time information; Performing traversal access on the first weight subgraph and determining a masking probability of the multiple groups of recognition results changing with time according to the information quantity weights of the multiple consecutive key frame data; the masking probability is used to describe the probability situation of the first weight subgraph being masked when accessing from one node in the first weight subgraph to the next node; Updating the first weight subgraph by using the masking probability and taking the updated first weight subgraph as the semantic representation result of the real-time vehicle data in the image modality.

4. The method according to claim 3, wherein The obtaining the information quantity weights of the multiple consecutive key frame data according to the multiple groups of recognition results includes: Obtaining the prior probabilities, confidence levels and recognition frame sizes respectively corresponding to the multiple groups of recognition results; Respectively determining the information quantity weights of the multiple consecutive key frame data according to the prior probabilities, confidence levels and recognition frame sizes respectively corresponding to the multiple groups of recognition results.

5. The method according to claim 1, characterized in that The real-time vehicle data in different modalities includes real-time vehicle data in a numerical modality; The performing semantic analysis on the real-time vehicle data in different modalities respectively to obtain semantic representation results of the real-time vehicle data in different modalities includes: Cut the vehicle real-time data of the numerical modality at a preset time interval to obtain at least two cutting results; Encode the at least two cutting results respectively, and use the obtained encoding results respectively as the semantic representation results of the vehicle real-time data of the numerical modality.

6. The method according to any one of claims 1 to 5, characterized in that The method further includes: Predict the vehicle working conditions according to the vehicle real-time data of different modalities after data integration.

7. The method according to any one of claims 1 to 5, characterized in that The method further includes: Identify the vehicle working conditions according to the vehicle real-time data of different modalities after data integration, and obtain the working condition identification results corresponding to the vehicle real-time data of different modalities after data integration respectively; Based on the working condition identification results corresponding to the vehicle real-time data of different modalities after data integration respectively, and based on the preset vehicle working condition labels associated with the vehicle data, calibrate the vehicle real-time data of different modalities after data integration respectively.

8. A data integration device, characterized in that, Includes: A vehicle data acquisition module, configured to acquire vehicle real-time data of different modalities during vehicle driving; A semantic representation result acquisition module, configured to perform semantic analysis on the vehicle real-time data of different modalities respectively to obtain the semantic representation results of the vehicle real-time data of different modalities; A classification module, configured to input the semantic representation results of the vehicle real-time data of different modalities into a classification model; The classification model is used to verify whether the semantic representation results of the vehicle real-time data of different modalities match each other; A data integration determination module, configured to determine that the data integration of the vehicle real-time data of different modalities has been completed when the result output by the classification model indicates that the semantic representation results of the vehicle real-time data of different modalities match. The data integration of the vehicle real-time data of different modalities has been completed, which is used to represent the semantic alignment of the vehicle real-time data of different modalities.

9. A data integration device, characterized in that, Includes a processor and a memory; the memory is used to store a computer program; the processor is used to execute the data integration method according to any one of claims 1-7 according to the computer program.

10. A computer-readable storage medium, characterized in that, For storing a computer program, the computer program, when run by a processor, executes the data integration method according to any one of claims 1-7.

Images