Remote early warning method and system based on big data and image recognition

Abstract

The invention provides a remote early warning method and system based on big data and image recognition, relates to the technical field of artificial intelligence, and is characterized in that a fusion perception neural network is used to carry out fusion perception operation on template vehicle networking driving data and template vehicle-mounted image data, and corresponding driving behavior description vectors and image semantic vectors are generated. Furthermore, the global comparison error of the fusion perception neural network is determined by performing feature space conversion on the driving behavior description vector and the image semantic vector and calculating the comparison error under different combinations. And optimizing the fusion perception neural network by using the global comparison error to generate an optimized network model. The optimized fusion perception neural network can efficiently and accurately carry out remote risk control early warning on any input candidate Internet of Vehicles driving data and candidate vehicle-mounted image data, the early warning accuracy and reliability are improved, and safe driving in the Internet of Vehicles environment is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a remote early warning method and system based on big data and image recognition. Background Art

[0002] With the rapid development and popularization of Internet of Vehicles technology, the amount of data generated during vehicle driving has exploded, and these data contain rich information about driving behavior and vehicle environment. At the same time, image recognition technology is also playing an increasingly important role in the fields of vehicle safety and intelligent driving. However, how to effectively use these big data and image information to remotely warn of potential risks during vehicle driving has become an urgent problem to be solved.

[0003] Traditional risk warning methods often rely on only a single driving data or image information, which is difficult to fully and accurately reflect the driving status and environmental conditions of the vehicle. In addition, these methods usually lack an effective fusion mechanism and cannot organically combine driving data and image information, thus limiting the accuracy and reliability of the warning. Summary of the invention

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a remote early warning method based on big data and image recognition, the method comprising: Based on the template Internet of Vehicles driving data and the template vehicle-mounted image data, a fusion perception operation is performed through a fusion perception neural network to generate a first driving behavior description vector of the template Internet of Vehicles driving data and a first image semantic vector of the template vehicle-mounted image data; Performing feature space conversion on the first driving behavior description vector and the first image semantic vector respectively to generate a second driving behavior description vector and a second image semantic vector; determining a first comparison error according to the first driving behavior description vector and the second image semantic vector, and determining a second comparison error according to the second driving behavior description vector and the first image semantic vector; Determining a global comparison error corresponding to the fused perception neural network according to the first comparison error and the second comparison error; The fused perception neural network is optimized according to the global comparison error to generate the optimized fused perception neural network, so as to perform remote risk control warning on any input candidate Internet of Vehicles driving data and candidate vehicle-mounted image data based on the optimized fused perception neural network.

[0005] On the other hand, an embodiment of the present invention also provides a remote warning system based on big data and image recognition, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0006] Based on the above aspects, the embodiment of the present application performs a fusion perception operation on the template Internet of Vehicles driving data and the template vehicle-mounted image data through a fusion perception neural network to generate corresponding driving behavior description vectors and image semantic vectors. Furthermore, by performing feature space conversion on the driving behavior description vector and the image semantic vector, and calculating the comparison errors under different combinations, the global comparison error of the fusion perception neural network is determined. The fusion perception neural network is optimized using the global comparison error to generate an optimized network model. The optimized fusion perception neural network can efficiently and accurately perform remote risk control warnings on any input candidate Internet of Vehicles driving data and candidate vehicle-mounted image data, thereby improving the accuracy and reliability of the warning, thereby contributing to safe driving in the Internet of Vehicles environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a schematic diagram of the execution flow of the remote early warning method based on big data and image recognition provided by an embodiment of the present invention.

[0008] Figure 2 It is a schematic diagram of the hardware architecture of a remote early warning system based on big data and image recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 The figure is a flow chart of a remote early warning method based on big data and image recognition provided by an embodiment of the present invention. The remote early warning method based on big data and image recognition is introduced in detail below.

[0010] Step S110, based on the template Internet of Vehicles driving data and the template vehicle-mounted image data, a fusion perception operation is performed through a fusion perception neural network to generate a first driving behavior description vector of the template Internet of Vehicles driving data and a first image semantic vector of the template vehicle-mounted image data.

[0011] In this embodiment, the template IoV driving data may be standard driving data collected from a large number of cars equipped with IoV devices, such as the speed, acceleration, steering angle, braking force and other information of the car recorded over a period of time. The template vehicle image data is image information captured by a camera inside or outside the car, such as the driver's facial expression, road conditions, the environment around the vehicle and other images.

[0012] For each data point in the template vehicle networking driving data, take the car speed as an example. Assuming that the speed of the car is 60 kilometers per hour at a certain moment, the fusion perception neural network first encodes the driving state according to the preset first driving state coding bitmap. This coding bitmap may specify the coding results corresponding to the speed in different intervals, for example, 0-50 kilometers per hour corresponds to code 1, 51-100 kilometers per hour corresponds to code 2, etc. Then the speed value of 60 kilometers per hour will be encoded as 2. Other driving state data such as the acceleration, steering angle, braking force, etc. of the car will also be encoded in a similar way to obtain the first state coding result of each driving state.

[0013] Then, a driving state fusion vector is generated based on the context information of each driving state in the template vehicle driving state data. For example, for the driving state of acceleration, its context information may include the speed change trend before this moment (if the speed has been stable at 50 kilometers per hour before, and suddenly accelerates to 60 kilometers per hour), whether the vehicle is on an uphill or downhill section (this will affect the relationship between acceleration and the actual driving state), the driver's driving habits (some drivers are used to rapid acceleration, while others are relatively stable), etc. The first state encoding result of acceleration is enhanced by using this context information. For example, if the context information shows that the vehicle is on an uphill slope and the driver is used to driving steadily, and the acceleration suddenly increases at this time, then the encoding result of this acceleration may be further adjusted to highlight this special situation.

[0014] Construct a state node mapping network, such as a multi-layer perceptron. Input the enhanced driving state representation set into the state node mapping network, and the output of the network is the driving state fusion vector. For each template vehicle driving state data, the driving state fusion vectors of multiple driving states are fused to finally generate a first driving behavior description vector. This first driving behavior description vector can comprehensively describe the driving behavior characteristics of the vehicle under the template vehicle network driving data.

[0015] For the template vehicle-mounted image data, take the driver's facial expression image as an example. The driving state is encoded for each image feature point in the image according to the second driving state coding bitmap. Assume that this coding bitmap specifies the coding results corresponding to different features of facial expressions, such as the degree of eye opening, the degree of eyebrow raising, the degree of curvature of the mouth corners, etc., all have their own coding ranges. For each image feature point in the image that represents a facial expression feature, calculate its second state coding result. Then, for each template vehicle-mounted image content, the second state coding result of each image feature point therein is fused to generate a first image semantic vector. This first image semantic vector can represent the semantic information in the template vehicle-mounted image data, for example, semantic information such as the driver's emotional state can be obtained from the driver's facial expression image.

[0016] Step S120 , performing feature space conversion on the first driving behavior description vector and the first image semantic vector respectively to generate a second driving behavior description vector and a second image semantic vector.

[0017] For example, a driving state fusion vector in the first driving behavior description vector may contain complex driving state information, such as speed, acceleration, and the result of fusion of these and context information. This driving state fusion vector is edited into numerical information to generate a second driving behavior description vector. Specifically, if this driving state fusion vector is originally represented by a complex structure, such as containing multiple levels of feature representation, then it is converted into a simple numerical form, such as converting it into a specific numerical sequence, which can represent the driving state information in a more compact and easier to process later.

[0018] For the first image semantic vector, it is assumed that the first image semantic vector is formed by encoding and fusion of image feature points in the template vehicle image content. It contains the semantic information of the image, but may have a high dimension. Taking the image of the surrounding environment of the car as an example, if the first image semantic vector contains the semantic coding information of multiple objects in the image (such as other vehicles, traffic signs, road conditions, etc.), the second state coding result corresponding to each image feature point is subjected to dimensionality reduction operation. For example, a dimensionality reduction algorithm such as principal component analysis (PCA) is used to convert the high-dimensional image feature point coding result into a low-dimensional representation to generate the second image semantic vector. The purpose of this is to reduce the complexity of the data while retaining the key information of the image semantics so that the data can be processed more efficiently in subsequent calculations.

[0019] Step S130: determining a first comparison error based on the first driving behavior description vector and the second image semantic vector, and determining a second comparison error based on the second driving behavior description vector and the first image semantic vector.

[0020] Continuing with the example of the automobile remote risk control warning system, we first calculate the actual feature distance between the template vehicle driving state data corresponding to the first driving behavior description vector and the template vehicle image content corresponding to the first image semantic vector. Assume that the template vehicle driving state data shows that the car is driving at a high speed and frequently changing lanes on a certain section of road, while the template vehicle image content shows that there are many vehicles on the road and the traffic signs are to slow down. From a practical point of view, this driving behavior is in conflict with the road conditions and traffic signs. The degree of this conflict is quantified through a specific algorithm, and this quantified value is the actual feature distance.

[0021] Determine a first feature distance between the first driving behavior description vector and the second image semantic vector. For example, after the first driving behavior description vector is transformed into a feature space, it represents the driving behavior characteristics of the car in a numerical form, while the second image semantic vector represents the semantic information of the image after dimensionality reduction processing. The distance between the two may be calculated using the Euclidean distance or other suitable distance measurement methods. Assume that in this scenario, the calculated first feature distance reflects the degree of difference between the driving behavior characteristics and the image semantic information in a certain feature space.

[0022] Then determine the second feature distance between the second driving behavior description vector and the first image semantic vector. Since the second driving behavior description vector and the first image semantic vector are the results of the conversion of the first driving behavior description vector and the second image semantic vector, the distance calculation method between them is similar to the above. The second feature distance obtained by using a specific distance measurement method also reflects the difference between the two in the feature space.

[0023] The first cost coefficient between the actual feature distance and the first feature distance is used as the first comparison error. The determination of this first cost coefficient may be based on a pre-set rule or obtained through model learning. For example, if the actual feature distance is large and the first feature distance is small, it means that the model has a certain degree of misestimation of the relationship between the driving behavior and the image semantics, then the first cost coefficient will be large, indicating that the first comparison error is large. Similarly, the first cost coefficient between the actual feature distance and the second feature distance is used as the second comparison error.

[0024] Step S140: determining a global comparison error corresponding to the fused perception neural network according to the first comparison error and the second comparison error.

[0025] In the scenario of the automobile remote wind control warning system, the first average error of each first comparison error is first determined. Assuming that there are multiple samples of template vehicle network driving data and template vehicle image data, the first comparison error is calculated for each sample. These first comparison errors are averaged to obtain the first average error. Similarly, the second average error of each second comparison error is determined.

[0026] For example, for a set of sample data, the first comparison errors calculated are 0.1, 0.2, 0.15, etc., then the first average error is (0.1+0.2+0.15) / 3=0.15. A similar calculation is performed for the second comparison error to obtain the second average error.

[0027] The weighted result of the first average error and the second average error is used as the weighted comparison error of the fusion perception neural network. This weighting coefficient may be set based on experience or the importance of the model. Assuming that the weighting coefficient of the first average error is 0.6 and the weighting coefficient of the second average error is 0.4, the weighted comparison error is 0.6*0.15+0.4*second average error.

[0028] Next, feature interaction is performed on the first driving behavior description vector and the first image semantic vector in the template vector combination. Taking dangerous scenes during car driving as an example, it is assumed that the first driving behavior description vector contains driving state fusion vectors such as sudden braking and sudden turning of the car, and the first image semantic vector contains the image feature point encoding of obstacles suddenly appearing in front. These two vectors are standardized, for example, their numerical ranges are adjusted to a specific interval. Then an association matrix is ​​constructed to calculate the similarity between each driving state fusion vector in the first driving behavior description vector and each image feature point encoding in the first image semantic vector. For example, the similarity between the driving state fusion vector of sudden braking and the image feature point encoding of an obstacle suddenly appearing in front may be high, and this similarity is used as an element of the association matrix.

[0029] Through this association matrix, the driving state fusion vector in the first driving behavior description vector is mapped to the image content space, and the image content space mapping vector corresponding to each driving state fusion vector is generated. For example, for the driving state fusion vector of sudden braking, the weighted sum is calculated according to its similarity with each image feature point code in the association matrix to obtain its mapping representation in the image content space. At the same time, the image feature point code in the first image semantic vector is mapped to the driving behavior space, and the driving behavior space mapping vector corresponding to each image feature point code is generated.

[0030] These mapping vectors are fused through a neural network fusion algorithm to generate a fused bidirectional mapping interaction vector group. This vector group contains the feature representation of driving behavior and image content that enhance each other. This vector group is input into one or more layers of neural network layers for nonlinear transformation and feature extraction to generate a hierarchically abstracted and refined target interaction feature representation. Finally, the target interaction feature representation is integrated through global average pooling or maximum pooling to generate the final feature interaction result.

[0031] Based on this feature interaction result, an estimate is made to generate a first confidence that the template vector combination is a valid template. For example, if the feature interaction result shows that the driving behavior is highly correlated with the image content and meets the characteristics of the car being in a dangerous scene, then the first confidence that the template vector combination is a valid template may be high. The annotation confidence that the template vector combination is a valid template is determined, and this annotation confidence may be determined based on manual annotation or known standard data.

[0032] Determine the third cost coefficient between the first confidence and the labeled confidence of each template vector combination, and use the weighted result of the third cost coefficient as the combined error of the fusion perception neural network. For example, for a template vector combination, the first confidence is 0.8 and the labeled confidence is 0.9, and the third cost coefficient may be calculated based on the difference between the two. Then, the third cost coefficients of all template vector combinations are weighted and summed to obtain the combined error.

[0033] Finally, the weighted comparison error and the combined error are fused to generate a global comparison error. For example, assuming the weighted comparison error is 0.2 and the combined error is 0.1, the global comparison error is 0.3 through a certain fusion rule (such as simple addition or weighted addition).

[0034] Step S150, optimizing the fused perception neural network according to the global comparison error, generating the optimized fused perception neural network, so as to perform remote risk control warning on any input candidate Internet of Vehicles driving data and candidate vehicle-mounted image data based on the optimized fused perception neural network.

[0035] In the remote risk control and early warning system for automobiles, the training data sequence is first obtained. This training data sequence contains multiple training data combinations, and each training data combination is associated with the annotation confidence corresponding to the template risk control and early warning event. For example, the training data combination can be a car in a specific driving state (such as speeding, fatigue driving, etc.) and the corresponding on-board image content (such as the driver yawning, the vehicle deviating from the lane, etc.), and the template risk control and early warning event is the corresponding risk event (such as increased risk of traffic accidents), and the annotation confidence is the degree of possibility of the event occurring determined based on actual conditions or expert knowledge.

[0036] Based on multiple training data combinations, the fusion perception neural network before optimization is used to estimate and generate the second confidence level corresponding to the template risk control warning event associated with each training data combination. For example, for the training data combination of speeding and the driver yawning, the fusion perception neural network may estimate that the second confidence level of a traffic accident in this case is 0.6.

[0037] According to the cost coefficient between each second confidence and the corresponding labeled confidence, the neuron weight information of the fusion perception neural network is optimized. For example, if the second confidence is 0.6 and the labeled confidence is 0.8, the cost coefficient between the two is large, indicating that there is a large deviation in the estimation of the neural network. According to this cost coefficient, the weight of the neuron is adjusted so that the neural network can more accurately estimate the confidence of the risk event in subsequent calculations. After multiple such adjustments, the optimized fusion perception neural network is generated.

[0038] When there is any input of candidate vehicle driving state data and candidate vehicle image content in the Internet of Vehicles big data of the target user, prediction is performed through the optimized fusion perception neural network. For example, the target user's car driving data shows that the car is driving at a higher speed at night, and the vehicle image content shows that the driver's vision is poor. These data are input into the optimized fusion perception neural network to generate the estimated confidence of the candidate risk control warning event matched by the candidate vehicle driving state data, such as the estimated confidence of the traffic accident risk is 0.7. According to this estimated confidence, the target risk control warning event matched by the candidate vehicle driving state data is determined. For example, when the estimated confidence exceeds a certain threshold (such as 0.5), it is judged as a high-risk event, triggering the corresponding remote risk control warning measures, such as sending a reminder message to the driver or notifying the insurance service platform and other relevant parties.

[0039] Therefore, in the automobile remote risk control warning system, the fusion perception neural network can be effectively used to process the Internet of Vehicles driving data and vehicle-mounted image data to achieve accurate remote risk control warning function.

[0040] Based on the above steps, the embodiment of the present application performs a fusion perception operation on the template Internet of Vehicles driving data and the template vehicle-mounted image data through a fusion perception neural network to generate corresponding driving behavior description vectors and image semantic vectors. Furthermore, by performing feature space conversion on the driving behavior description vector and the image semantic vector, and calculating the comparison errors under different combinations, the global comparison error of the fusion perception neural network is determined. The fusion perception neural network is optimized using the global comparison error to generate an optimized network model. The optimized fusion perception neural network can efficiently and accurately perform remote risk control warnings on any input candidate Internet of Vehicles driving data and candidate vehicle-mounted image data, thereby improving the accuracy and reliability of the warning, thereby contributing to safe driving in the Internet of Vehicles environment.

[0041] In a possible implementation, step S130 may include: The actual feature distance between the following feature information is calculated: the template vehicle driving state data corresponding to the first driving behavior description vector and the template vehicle image content corresponding to the first image semantic vector.

[0042] A first characteristic distance between the first driving behavior description vector and the second image semantic vector is determined, and a second characteristic distance between the second driving behavior description vector and the first image semantic vector is determined.

[0043] A first cost coefficient between the actual feature distance and the first feature distance is used as the first comparison error, and a first cost coefficient between the actual feature distance and the second feature distance is used as the second comparison error.

[0044] In this embodiment, the actual feature distance between the template vehicle driving state data corresponding to the first driving behavior description vector and the template vehicle image content corresponding to the first image semantic vector is first calculated. For example, the template vehicle driving state data contains a series of data such as the driving speed of the car on a specific road section is 80 kilometers per hour, the acceleration is 2 meters per second per second, the steering angle is 5 degrees, and the driver has frequent lane changes. The template vehicle image content corresponding to the first image semantic vector shows that the road section is densely populated, and the road sign shows that the maximum speed limit is 60 kilometers per hour and there is a curve sign. In order to calculate the actual feature distance, it is necessary to comprehensively consider these different types of data. The driving state data such as speed, acceleration, and steering angle can be quantitatively compared with the image content such as road conditions and traffic signs. For example, the difference between the speed exceeding the speed limit and the speed limit of the traffic sign can be quantified by the speed difference, and the degree of conflict between frequent lane changes and dense vehicle road conditions can be quantified by the set rules. These quantified values ​​are combined to obtain the actual feature distance.

[0045] Then, the first characteristic distance between the first driving behavior description vector and the second image semantic vector is determined. The first driving behavior description vector is a vector that can comprehensively describe the driving behavior after the fusion perception operation of the template vehicle driving state data, and the second image semantic vector is a vector after the feature space conversion of the first image semantic vector. It is assumed that the first driving behavior description vector represents the comprehensive characteristics of the driving state in a numerical way, such as representing the comprehensive influence of factors such as speed, acceleration, steering angle and driver behavior in a specific numerical sequence. The second image semantic vector represents the image semantic information in a compact form after operations such as dimensionality reduction. When determining the first characteristic distance, a suitable distance measurement method is used, such as the Euclidean distance calculation method. Each element in the first driving behavior description vector is calculated with the corresponding element in the second image semantic vector to obtain the sum of the squares of the differences of each element, and then the square root is taken to obtain the first characteristic distance. This distance reflects the degree of difference between the converted driving behavior characteristics and the image semantic information in a specific feature space.

[0046] Then determine the second feature distance between the second driving behavior description vector and the first image semantic vector. The second driving behavior description vector is a vector obtained by performing feature space transformation on the first driving behavior description vector, and the first image semantic vector is a vector obtained by performing fusion perception operation on the template vehicle image content. A suitable distance measurement method is also used to determine the second feature distance. For example, the second driving behavior description vector represents the driving behavior in an edited numerical information, and the first image semantic vector represents the image semantics in a fusion image feature point encoding manner. According to the distance measurement rule, the elements in the two vectors are calculated to obtain the second feature distance, which also reflects the difference between the two in the feature space.

[0047] Finally, the first cost coefficient between the actual feature distance and the first feature distance is used as the first comparison error, and the first cost coefficient between the actual feature distance and the second feature distance is used as the second comparison error. The determination of the first cost coefficient is based on a pre-set rule or obtained through model learning. For example, if the actual feature distance is large, it indicates that there is a large difference between the actual driving state data and the vehicle-mounted image content, and the first feature distance is small, which means that the difference between the driving behavior characteristics after feature conversion and the image semantic information is small. In this case, it may be explained that the model is unreasonable in processing the relationship between driving behavior and image semantics, and the first cost coefficient will be large, indicating that the first comparison error is large. Similarly, for the second comparison error, the first cost coefficient is determined according to the same rule based on the actual feature distance and the second feature distance, so as to obtain the second comparison error. In this way, the error between driving behavior and image semantics in different conversion and processing processes can be accurately quantified, providing a basis for the optimization of the subsequent fusion perception neural network.

[0048] In a possible implementation, step S140 may include: Step S141, determining the first average error of each of the first comparison errors, and determining the second average error of each of the second comparison errors, and taking the weighted result of the first average error and the second average error as the weighted comparison error of the fused perceptual neural network.

[0049] In this embodiment, during the collection of automobile driving data, there will be multiple samples of template vehicle driving state data and template vehicle-mounted image content. For each sample, the first comparison error and the second comparison error are calculated according to the method described above. For example, assuming that there are 10 samples, the first comparison errors calculated for each sample are 0.1, 0.15, 0.12, 0.08, 0.11, 0.13, 0.09, 0.14, 0.1, and 0.12, respectively. These first comparison errors are added and divided by the number of samples 10, that is, (0.1+0.15+0.12+0.08+0.11+0.13+0.09+0.14+0.1+0.12) / 10=0.114, and the first average error is obtained. Similarly, for the second comparison error, assuming that the calculated values ​​are 0.1, 0.13, 0.11, 0.09, 0.12, 0.14, 0.1, 0.13, 0.11, 0.12, the second average error is (0.1+0.13+0.11+0.09+0.12+0.14+0.1+0.13+0.11+0.12) / 10=0.115. The weighted coefficient of the first average error is set to 0.6, and the weighted coefficient of the second average error is set to 0.4, then the weighted comparison error is 0.6×0.114+0.4×0.115=0.1144.

[0050] Step S142: performing feature interaction on the first driving behavior description vector and the first image semantic vector in the template vector combination to generate a feature interaction result.

[0051] Step S143, making an estimate based on the feature interaction result, generating a first confidence level that the template vector combination is a valid template, wherein when the template vehicle driving state data corresponding to the first driving behavior description vector and the template vehicle image content corresponding to the first image semantic vector reflect the same risk control warning event, the template vector combination is a valid template.

[0052] In this embodiment, for example, when the template vehicle driving status data shows that the car is speeding and frequently changing lanes, and the template vehicle image content corresponding to the first image semantic vector shows that the road is narrow and the vehicles are dense, both of which reflect the risk control warning event of increased traffic accident risk, then this template vector combination is a valid template. According to the feature information in the feature interaction result, the first confidence that this template vector combination is a valid template is estimated through a pre-trained model or set rules. Assuming that certain feature values ​​in the feature interaction result indicate that the relationship between this driving behavior and the image content is highly correlated with the increased risk of traffic accidents, then the first confidence may be high, such as 0.8.

[0053] Step S144, determining the annotation confidence that the template vector combination is a valid template, determining a third cost coefficient between the first confidence and the annotation confidence of each template vector combination, and using the weighted result of the third cost coefficient as the combination error of the fusion perception neural network.

[0054] In this embodiment, the annotation confidence is obtained based on known accurate information or manual annotation. For example, for the above template vector combination reflecting the increased risk of traffic accidents, through the analysis of a large amount of historical data or the judgment of experts, it is determined that the annotation confidence of this template vector combination that does reflect the increased risk of traffic accidents is 0.9.

[0055] For each template vector combination, the difference between the first confidence and the labeled confidence is calculated, and then the third cost coefficient is calculated according to the pre-set rules based on the difference. For example, if the first confidence is 0.8, the labeled confidence is 0.9, and the difference is 0.1, the third cost coefficient calculated according to the set rules is 0.2. Assuming that there are multiple template vector combinations, the third cost coefficient is calculated for each template vector combination, and then these third cost coefficients are weighted and summed according to a certain weighting method (such as setting different weighting coefficients according to the importance of the template vector combination) to obtain the combined error of the fusion perception neural network.

[0056] Step S145 , fusing the weighted comparison error and the combined error to generate the global comparison error.

[0057] For example, the weighted comparison error is 0.1144, and the combined error is 0.15. By simply adding or merging the two according to certain weighted rules (such as weighted addition, setting the weighting coefficients to 0.5), the global comparison error is 0.2644 or other results are obtained by weighted addition. This global comparison error can fully reflect the overall error of the fusion perception neural network in processing driving behavior and image semantic relationships, and provide an accurate basis for the subsequent optimization of the fusion perception neural network.

[0058] In a possible implementation, step S142 includes: Step S1421, obtain a first driving behavior description vector and a first image semantic vector generated by a fusion perception neural network, wherein the first driving behavior description vector includes multiple driving state fusion vectors, each driving state fusion vector is a comprehensive representation of each driving state in the template vehicle driving state data after encoding and context information mapping, and the first image semantic vector is formed by encoding and fusion of image feature points in the template vehicle-mounted image content.

[0059] The first driving behavior description vector includes multiple driving state fusion vectors. For example, during the driving process of the car, one of the driving state fusion vectors is a comprehensive representation of the car's speed and acceleration. It is obtained by encoding the two driving states of speed and acceleration in the driving state data of the template vehicle and mapping them with context information. The speed encoding may set different encoding values ​​based on different speed ranges, such as 0-50 km / h is encoded as 1, 51-100 km / h is encoded as 2, etc., and acceleration also has a similar encoding method. The context information of acceleration may include factors such as the slope of the road and the load of the vehicle. These factors are mapped together with the encoding of acceleration to obtain the driving state fusion vector. The first image semantic vector is obtained by encoding and fusion of image feature points in the template vehicle image content. For example, image feature points such as traffic signs on the road, the position of other vehicles, and the driver's facial expression in the vehicle image have their own encodings. These encodings are fused to form the first image semantic vector.

[0060] Step S1422: normalize the first driving behavior description vector and the first image semantic vector to generate a standardized first driving behavior description vector and a standardized first image semantic vector.

[0061] The purpose of normalization is to adjust the numerical range of these vectors to a suitable interval for subsequent calculations. For example, for each element in the first driving behavior description vector, a linear transformation is used to map its value to between 0 and 1. If the value of an element in a driving state fusion vector is originally between -10 and 10, it is mapped to between 0 and 1 by calculating (element value + 10) / 20. A similar normalization method is also used for the first image semantic vector.

[0062] Step S1423, based on the standardized first driving behavior description vector and the standardized first image semantic vector, construct an association matrix, the association matrix is ​​used to represent the potential connection between driving behavior features and image semantic features. The construction process includes: calculating the similarity between each driving state fusion vector in the first driving behavior description vector and each image feature point code in the first image semantic vector, and using the calculated similarity as an element of the association matrix, the row of the association matrix corresponds to the driving state fusion vector, the column of the association matrix corresponds to the image feature point code, and generating an association matrix representing the strength of association between driving behavior and image content.

[0063] For example, a driving state fusion vector is a comprehensive representation of the sudden braking of a car, and there is a corresponding numerical representation in the standardized vector. An image feature point encoding is a representation of an obstacle suddenly appearing in front of the vehicle image. The similarity between the two is calculated by a similarity calculation method (such as cosine similarity). If there is a strong correlation between the sudden braking and the sudden appearance of an obstacle in front, the calculated similarity may be high. This similarity value is used as an element in the association matrix, where the rows of the association matrix correspond to the driving state fusion vectors and the columns correspond to the image feature point encodings. In this way, an association matrix representing the strength of the association between driving behavior and image content is generated.

[0064] Step S1424, using the association matrix, maps the driving state fusion vector in the first driving behavior description vector to the image content space, and generates an image content space mapping vector corresponding to each driving state fusion vector. The specific operation is: for each driving state fusion vector, according to the similarity between the driving state fusion vector and each image feature point encoding in the association matrix, calculate the weighted sum, wherein the weighted weight is the similarity value, and the result of the weighted sum is used as the mapping representation of the driving state in the image content space.

[0065] For example, for the driving state fusion vector of sudden braking, it has different similarities with different image feature point codes (such as image feature point codes corresponding to the vehicle in front, traffic signs, road conditions, etc.). These similarities are used as weighted weights to perform weighted summation on the corresponding image feature point codes. The result of the weighted sum is the mapping representation of the driving state of sudden braking in the image content space.

[0066] Step S1425, maps the image feature point codes in the first image semantic vector to the driving behavior space, and generates a driving behavior space mapping vector corresponding to each image feature point code. The specific operation is: for each image feature point code, calculate the weighted sum according to the similarity between the image feature point code and each driving state fusion vector in the association matrix, and obtain the mapping representation of the image feature point in the driving behavior space.

[0067] For example, for the image feature point encoding of an obstacle suddenly appearing in front, it has different similarities with the fusion vectors of different driving states (such as sudden braking, turning, etc.). A weighted sum is performed according to the similarity to obtain the mapping representation of this image feature point in the driving behavior space.

[0068] Step S1426, through a neural network fusion algorithm, the image content space mapping vector of each driving state fusion vector is fused with the driving behavior space reflection vector encoded by each image feature point to generate a fused bidirectional mapping interaction vector group, wherein the fused bidirectional mapping interaction vector group contains feature representations of mutual enhancement of driving behavior and image content.

[0069] For example, if the image content space mapping vector of a driving state fusion vector represents the association between a car's sudden braking and an obstacle ahead, and a driving behavior space reflection vector encoded by an image feature point represents the association between the obstacle ahead and the car's sudden braking, these two vectors are fused through a neural network fusion algorithm. The fused vector can more comprehensively represent the relationship between sudden braking and the obstacle ahead, and this relationship includes the mutual influence of driving behavior and image content.

[0070] Step S1427, input the fused bidirectional mapping interaction vector group into one or more levels of neural network layers for nonlinear transformation and feature extraction, and generate a hierarchical abstract and refined target interaction feature representation.

[0071] In the neural network layer, the input vector is processed through nonlinear transformation by activation functions (such as ReLU function) to extract higher-level feature representations. For example, the first neural network layer may extract preliminary features about the relationship between driving behavior and image content. As the layers increase, subsequent neural network layers will further abstract and refine these features, and finally obtain the target interaction feature representation after hierarchical abstraction and refinement.

[0072] Step S1428, integrating the target interaction feature representations through global average pooling or maximum pooling to generate a final feature interaction result.

[0073] For example, if the global average pooling method is used, all elements in the target interaction feature representation are averaged to obtain a value representing the overall feature interaction result. If the maximum pooling method is used, the maximum value in the target interaction feature representation is selected as the final feature interaction result. This feature interaction result reflects the deep interaction information between the first driving behavior description vector and the first image semantic vector.

[0074] In a possible implementation, step S110 includes: Step S111, by fusing the perception neural network, according to the first driving state coding bitmap, the driving state data of each template vehicle in the template Internet of Vehicles driving data is encoded to generate a first state coding result for each driving state in each of the template vehicle driving state data, the template vehicle driving state data includes multiple driving states, and the first driving state coding bitmap includes a coding conversion relationship between the driving state and the first state coding result.

[0075] In this embodiment, the template vehicle driving state data includes multiple driving states, such as the speed, acceleration, steering angle, braking force and other information of the car, which are key factors for describing the driving state of the car. The first driving state coding bitmap includes the coding conversion relationship between the driving state and the first state coding result, which specifies a specific coding method for each driving state. Taking the speed of the car as an example, it is assumed that the first driving state coding bitmap stipulates that the speed is coded as 1 in the range of 0-50 kilometers / hour, 51-100 kilometers / hour is coded as 2, 101-150 kilometers / hour is coded as 3, etc. When the speed of the car in the template vehicle network driving data is 70 kilometers / hour, according to this coding bitmap, its first state coding result is 2. For the acceleration of the car, there are similar coding rules, such as the acceleration is coded as 1 in -2-0 meters / second squared, coded as 2 in 0-2 meters / second squared, etc., and its first state coding result is determined according to the actual acceleration value. In this way, each driving state in the template vehicle driving state data is encoded to obtain the first state coding result of each driving state.

[0076] Step S112, based on the context information of each driving state in the template vehicle driving state data, performing state node mapping on the first state encoding result of each driving state to generate a driving state fusion vector of each driving state.

[0077] Step S113: for each of the template vehicle driving state data, the driving state fusion vectors of the multiple driving states included in the template vehicle driving state data are fused to generate a first driving behavior description vector.

[0078] For example, the driving state data of the template vehicle includes three driving states: speed, acceleration, and steering angle. Their respective driving state fusion vectors are [0.4, 0.35, 0.25], [0.3, 0.4, 0.3], and [0.2, 0.3, 0.5]. These three vectors are fused into a first driving behavior description vector through a specific fusion method (such as weighted summation or splicing). If the weighted summation method is used, assuming that the weighted coefficients of speed, acceleration, and steering angle are 0.4, 0.3, and 0.3 respectively, then the first element of the first driving behavior description vector is 0.4×0.4+0.3×0.3+0.3×0.2=0.31. The other elements are calculated in the same way to finally obtain the first driving behavior description vector, which can comprehensively describe the driving behavior characteristics of the template vehicle under the driving data of the Internet of Vehicles.

[0079] Step S114, based on the second driving state coding bitmap, the driving state of each template vehicle image content in the template vehicle image data is encoded to generate a second state coding result for each image feature point in each template vehicle image content, and the second driving state coding bitmap includes the coding conversion relationship between the image feature point and the second state coding result.

[0080] The template vehicle image content contains various information, such as the driver's facial expression, the position of other vehicles around the vehicle, the road conditions and other image features. The second driving state coding bitmap includes the coding conversion relationship between the image feature point and the second state coding result. Taking the driver's facial expression as an example, it is assumed that the second driving state coding bitmap stipulates that the degree of eye opening is coded as 1 within a certain range, and coded as 2 beyond this range; the degree of eyebrow raising is coded as 1 within a certain interval, and coded as 2 outside this interval, etc. For the image feature point of the driver's facial expression in the vehicle image, according to the characteristics such as the degree of eye opening and eyebrow raising, its second state coding result is determined according to the coding bitmap. For the image feature point of the position of other vehicles around the vehicle, it can also be coded according to factors such as the distance between it and the vehicle, such as coded as 1 if the distance is 0-5 meters, coded as 2 if the distance is 5-10 meters, etc. In this way, each image feature point in the template vehicle image content is encoded to obtain the second state coding result of each image feature point.

[0081] Step S115 , for each of the template vehicle-mounted image contents, the second state encoding result of each of the image feature points in the template vehicle-mounted image contents is fused to generate a first image semantic vector.

[0082] For example, the template vehicle image content contains three image feature points: the driver's facial expression, other vehicle positions, and road conditions. Their respective second-state encoding results are [1, 2], [2, 1], and [1, 1] (this is just a simple example, and the actual encoding results may be more complex). These results are fused into a first image semantic vector through a specific fusion method (such as splicing these encoding results in a certain order or performing weighted summation, etc.). If the splicing method is used, the first image semantic vector obtained may be [1, 2, 2, 1, 1, 1]. This vector can represent the semantic information in the template vehicle image data. For example, semantic information such as the driver's facial expression state and the environmental conditions around the vehicle can be obtained from this vector, thereby providing an image basis for subsequent remote risk control warnings of automobiles.

[0083] Through the above steps, the template Internet of Vehicles driving data and template vehicle image data are processed using a fusion perception neural network to generate the first driving behavior description vector and the first image semantic vector. These two vectors provide the key data basis for the automobile remote risk control warning system from the perspectives of driving behavior and image semantics respectively.

[0084] In a possible implementation, step S112 includes: Step S1121, for each driving state in the template vehicle driving state data, extract the context information corresponding to the driving state, the context information includes the previous and next adjacent states of the driving state in the time series, the vehicle dynamic parameters associated with the driving state, the environmental perception data and the driving behavior mode identification.

[0085] Step S1122: utilizing the context information to enhance the representation of the first state encoding result of each driving state, and generating an enhanced driving state representation set.

[0086] For example, for the driving state of the car speed, its previous and next adjacent states in the time series are important context information. If the car speed has been stable at 50 kilometers per hour before, and now it changes to 70 kilometers per hour, this speed change trend is part of the context information. Vehicle dynamic parameters associated with speed are also included in the context information, such as the vehicle load. If the vehicle load is large, the speed change will be different under the same power output. Environmental perception data is also a component of context information. For example, factors such as the slope of the road and the friction of the road surface will affect the speed of the car. In addition, the driving behavior mode identification also needs to be considered, such as whether the driver is driving aggressively or steadily. If it is an aggressive driving mode, the speed may change more frequently and with a larger amplitude. Using these context information, the first state encoding result of the driving state of speed is enhanced to generate an enhanced driving state representation set. For example, if the first state encoding result of speed is 2, after combining the context information, this encoding result may be adjusted or some additional identifications may be added to more comprehensively represent the driving state of speed. The same operation is performed for other driving states such as acceleration, steering angle, etc. to obtain respective enhanced driving state representation sets.

[0087] Step S1123, construct a state node mapping network, the state node mapping network is used to map the enhanced driving state representation set to the driving state fusion vector space, the state node mapping network is a deep learning model, including a multi-layer perceptron, a convolutional neural network or a graph neural network. The input of the state node mapping network is the enhanced driving state representation, and the output is the driving state fusion vector. During the training process of the state node mapping network, a set of labeled driving state data is used as a training set, wherein each driving state corresponds to a predefined driving state fusion vector, and the network parameters of the state node mapping network are optimized by minimizing the difference between the output of the state node mapping network and the target fusion vector, so that the state node mapping network maps the enhanced driving state representation to the driving state fusion vector space.

[0088] Step S1124: Map the enhanced driving state representation set using the state node mapping network to generate a driving state fusion vector for each driving state.

[0089] Taking the multilayer perceptron as an example, its input is the enhanced driving state representation, and its output is the driving state fusion vector. In the training process of the state node mapping network, a set of labeled driving state data is used as the training set, in which each driving state corresponds to a predefined driving state fusion vector. For example, for the driving state of 70 km / h, medium load, flat road slope, and steady driving mode, the corresponding driving state fusion vector is predefined as [0.5, 0.3, 0.2] (this is just an example, and the actual fusion vector may contain more elements to fully represent the driving state). The network parameters of the state node mapping network are optimized by minimizing the difference between the output of the state node mapping network and the target fusion vector. In the training process, the enhanced driving state representation is input into the state node mapping network to obtain the output driving state fusion vector, and the difference between this output and the target fusion vector is calculated, such as using the mean square error (MSE) as the difference measurement indicator. According to this difference, the network weight and other parameters are adjusted. After multiple iterative training, the state node mapping network can accurately map the enhanced driving state representation to the driving state fusion vector space. The enhanced driving state representation set is mapped using the trained state node mapping network to generate a driving state fusion vector for each driving state. For example, for the driving state of speed, after being mapped by the state node mapping network, its driving state fusion vector is [0.4, 0.35, 0.25]. This driving state fusion vector can more comprehensively represent the comprehensive situation of the driving state of speed under the influence of various factors.

[0090] In a possible implementation, step S120 may include: Step S121: Edit each driving state fusion vector in the first driving behavior description vector into numerical information to generate the second driving behavior description vector.

[0091] Step S122: performing dimension reduction on the second state encoding result corresponding to each image feature point in the first image semantic vector to generate the second image semantic vector.

[0092] In this embodiment, the first driving behavior description vector includes multiple driving state fusion vectors, which are obtained through a series of complex operations and comprehensively reflect various state information of the car during driving. Each driving state fusion vector in the first driving behavior description vector is edited into numerical information to generate a second driving behavior description vector. For example, a driving state fusion vector in the first driving behavior description vector may be a representation form after comprehensively considering context information about driving states such as car speed, acceleration and steering angle, and this representation form may include a combination of multiple data structures or encoding methods. In order to convert it into numerical information, it is necessary to re-edit each element in this driving state fusion vector. Assume that the speed representation part in this driving state fusion vector is obtained according to a certain interval division and encoding rule, such as the speed is coded as 2 at 50-70 kilometers per hour, and this code also has a certain association representation with context information such as speed change trend and vehicle load. When editing into numerical information, it is necessary to convert this complex representation into a pure numerical form. According to the pre-set conversion rules, the speed code 2 and the context information related to the speed can be converted into a specific value, such as 1.5, according to a certain algorithm (this value is calculated based on the speed code, speed change trend, vehicle load and other factors). The acceleration, steering angle and other parts in the driving state fusion vector are also converted in a similar way, and the entire driving state fusion vector is edited into a numerical sequence, which is a part of the second driving behavior description vector. By performing such an editing operation on each driving state fusion vector in the first driving behavior description vector, a complete second driving behavior description vector is finally obtained. This second driving behavior description vector represents the driving behavior characteristics of the car in a more concise and easier to process numerical form, so that this information can be used more efficiently in subsequent calculations and analysis for operations related to remote risk control warning of the car.

[0093] The first image semantic vector is formed by encoding and fusion of the image feature points in the template vehicle image content, wherein the second state encoding result corresponding to each image feature point contains rich image semantic information, but may have a high dimension. The second state encoding result corresponding to each image feature point in the first image semantic vector is subjected to a dimensionality reduction operation to generate the second image semantic vector. For example, in the first image semantic vector, the second state encoding result corresponding to the image feature point of the driver's facial expression may contain information of multiple dimensions such as the degree of eye opening and closing, the degree of eyebrow raising, and the degree of curvature of the mouth corners. This information exists in a multi-dimensional encoding form in the first image semantic vector. Since such high-dimensional information may bring a large computational burden in subsequent calculations and analysis, and there may be some redundant information, a dimensionality reduction operation is required. The principal component analysis (PCA) algorithm is used for dimensionality reduction. First, the covariance matrix between the dimensions is calculated. This covariance matrix reflects the correlation between different dimensions. Then, the covariance matrix is ​​subjected to eigenvalue decomposition to obtain eigenvalues ​​and eigenvectors. According to the size of the eigenvalue, the first few main eigenvectors are selected. The dimensions corresponding to these eigenvectors can retain the information of the original data to the greatest extent. The second state encoding result corresponding to each image feature point is projected into the space formed by these main feature vectors to obtain the result after dimensionality reduction. For example, the second state encoding result corresponding to the image feature point of the driver's facial expression is a 5-dimensional vector, which may become a 2-dimensional vector after dimensionality reduction. Such dimensionality reduction operation is performed on the second state encoding result corresponding to each image feature point in the first image semantic vector. The final second image semantic vector retains the key information of image semantics while reducing the dimension of the data, improving the efficiency of data processing, and is more conducive to subsequent calculations and analysis in the remote risk control warning system of the vehicle, such as comparison and fusion with the driving behavior description vector, so as to more accurately judge the risk status of the vehicle during driving.

[0094] In a possible implementation, step S150 includes: Step S151, obtaining a training data sequence, wherein the training data sequence includes multiple training data combinations and the annotation confidence corresponding to the template risk control warning event associated with each training data combination, and the training data combination includes: template vehicle driving status data and template vehicle image content of the template risk control warning event.

[0095] In this embodiment, the template vehicle driving state data covers various key information during the driving process of the car, such as the driving state parameters of the car such as the speed, acceleration, steering angle, and braking force. Taking speed as an example, it can be a record of different speed values ​​of the car on a specific road section, such as 50 kilometers per hour, 60 kilometers per hour, etc. Acceleration can be a specific value of the vehicle during the start, acceleration, and deceleration process, such as 2 meters per square second, -1 meter per square second, etc. The steering angle may include the angle value of the vehicle when turning, such as 15 degrees, 30 degrees, etc. The braking force also has a corresponding numerical record, indicating the intensity when braking. The template vehicle-mounted image content contains image-related information obtained from the internal or external camera of the car, such as the driver's facial expression (whether the eyes are tired, whether the expression is nervous, etc.), the traffic conditions around the vehicle (the number, distance, relative speed, etc. of other vehicles on the road), road signs (speed limit signs, curve signs, etc.), etc. The annotation confidence corresponding to the template risk control warning event associated with each training data combination is determined based on historical data, expert experience or actual accident statistics, etc., indicating the degree of possibility of a specific template risk control warning event occurring under given template vehicle driving status data and template vehicle image content. For example, when the template vehicle driving status data shows that the car is speeding and the template vehicle image content shows that the driver has a nervous expression, based on previous accident data and experience, it may be determined that the risk of a traffic accident in this case increases. The annotation confidence of this template risk control warning event is 0.8, indicating that there is a high possibility of a risk event occurring.

[0096] Step S152: Based on the multiple training data combinations, the optimized fusion perception neural network is used to estimate and generate a second confidence level corresponding to the template risk control warning event associated with each training data combination.

[0097] In this embodiment, the optimized fusion perception neural network has been optimized in the previous steps and can better process the input training data combination. For each training data combination, its template vehicle driving state data and template vehicle image content are input into the optimized fusion perception neural network. For example, the template vehicle driving state data of a training data combination is a car speed of 80 kilometers per hour (speeding), large acceleration and frequent changes in steering angle, and the template vehicle image content is that the driver's eyes are half closed (signs of fatigue driving) and there are many vehicles on the road. The neurons in the network will perform complex calculations and processing on these input data. During the forward propagation of the network, the neurons calculate the input data according to the pre-learned weights and activation functions. For speed information, some neurons in the network may perform weighted processing on the value of 80 kilometers per hour, and fuse the neuron calculation results related to other driving state data (such as acceleration and steering angle) and image content (signs of driver fatigue driving and road vehicle conditions). After calculation by multiple layers of neurons, a result is finally output, which is the second confidence corresponding to the template risk control warning event associated with the training data combination. Assume that for this training data combination, the second confidence level of the fusion perception neural network output is 0.7, which means that under this input condition, according to the network's estimation, the probability of this template risk control warning event increasing the risk of a traffic accident is 0.7.

[0098] Step S153, optimizing the neuron weight information of the fused perceptual neural network according to the cost coefficient between each of the second confidences and the corresponding labeled confidences, and generating the optimized fused perceptual neural network.

[0099] In this embodiment, for each training data combination, the difference between the second confidence and the labeled confidence is calculated. For example, in the aforementioned training data combination, the second confidence is 0.7, the labeled confidence is 0.8, and there is a difference of 0.1 between the two. This difference will be used to calculate the cost coefficient, and the calculation method of the cost coefficient is based on a pre-set rule, which may be based on factors such as the size of the difference and the importance of the data. Assume that the cost coefficient calculated based on this difference is 0.2. Then adjust the neuron weight information of the fusion perception neural network according to this cost coefficient. In a neural network, the weight of the neuron determines the degree of influence of the input data on the output result. When a difference is found between the second confidence and the labeled confidence, it means that the current weight setting of the network may not be optimal. Through the back propagation algorithm, adjust the weight of the neuron according to the cost coefficient. For example, if a neuron has too high a weight for the input data of speed, resulting in a large difference between the second confidence and the labeled confidence, then in the back propagation process, the weight of the neuron for the speed data will be reduced. After calculation and weight adjustment of all training data combinations, an optimized fusion perception neural network is generated. This fusion perception neural network can estimate the input data more accurately and reduce the difference between the second confidence and the annotation confidence.

[0100] Step S154, predicting the candidate vehicle driving status data and candidate vehicle-mounted image content in the Internet of Vehicles big data of any input target user through the optimized fusion perception neural network, generating an estimated confidence of the candidate risk control warning event matching the candidate vehicle driving status data, and determining the target risk control warning event matching the candidate vehicle driving status data based on the estimated confidence of the candidate risk control warning event matching the candidate vehicle driving status data.

[0101] In this embodiment, the Internet of Vehicles big data of the target user contains rich information, wherein the candidate vehicle driving state data includes various state information of the car during actual driving, such as speed, acceleration, steering angle, etc., and the candidate vehicle image content is the image information captured during the driving process of the car, such as the driver's state, road conditions, etc. These candidate vehicle driving state data and candidate vehicle image content are input into the optimized fusion perception neural network. For example, one of the candidate vehicle driving state data in the Internet of Vehicles big data of the target user is that the car is driving at a speed of 100 kilometers per hour at night (speeding), the acceleration is unstable, and the candidate vehicle image content shows that the driver's vision is poor and there is fog on the road. The fusion perception neural network processes these input data, and after calculation and fusion of neurons, it finally outputs the estimated confidence of the candidate risk control warning event (such as increased risk of traffic accidents) matched by the candidate vehicle driving state data, assuming that this estimated confidence is 0.9. According to this estimated confidence, the target risk control warning event matched by the candidate vehicle driving state data is determined. If a threshold is set to 0.8, when the estimated confidence is greater than or equal to this threshold, the target risk control warning event matched by the candidate vehicle driving status data is judged to be an increased risk of traffic accidents. At this time, the system can take corresponding measures, such as sending a reminder message to the driver or notifying the insurance service platform and other relevant parties.

[0102] In a possible implementation, the method further includes: Step S160, optimizing the product content of each insurance service product subscribed by the target user according to each target risk control warning event of the target user.

[0103] In this embodiment. After determining each target risk control warning event of the target user, for example, the target risk control warning event of the target user having a higher risk of increased traffic accidents, the insurance service platform can optimize the product content of the automobile insurance service product subscribed by the target user according to this information. If the target user is often in a high-risk driving state (such as speeding, fatigue driving, etc.), the insurance service platform may adjust the insurance terms, increase the premium or adjust the scope of insurance liability. For example, for target users who are prone to traffic accident risks, the insurance service platform may increase the premium of their third-party liability insurance, and may limit the scope of claims in certain high-risk driving situations. Alternatively, if the target user has good driving behavior for a period of time and no high-risk target risk control warning events occur, the insurance service platform can give certain preferential policies, such as reducing premiums or increasing insurance service content (such as increasing the number of road rescue services, etc.). In this way, the insurance service platform can formulate insurance product content more accurately according to the actual risk situation of the target user, achieve a balance between risk and benefit, and also encourage the target user to improve driving behavior and reduce the occurrence of risk events.

[0104] Based on the above steps, not only can we accurately issue risk warnings for the target users' Internet of Vehicles big data, but we can also optimize the content of insurance service products based on the risk warning results, thereby improving the risk management efficiency and service quality of the entire automobile insurance industry.

[0105] Figure 2 The hardware structure of the remote warning system 100 based on big data and image recognition for implementing the remote warning method based on big data and image recognition provided by an embodiment of the present invention is shown as follows: Figure 2 As shown, the remote warning system 100 based on big data and image recognition may include a processor 110 , a machine-readable storage medium 120 , a bus 130 , and a communication unit 140 .

[0106] The machine-readable storage medium 120 may store data and / or instructions. In some embodiments, the machine-readable storage medium 120 may store data acquired from an external terminal. In some embodiments, the machine-readable storage medium 120 may store data and / or instructions that the remote early warning system 100 based on big data and image recognition uses to execute or use to complete the exemplary method described in the present invention.

[0107] During the specific implementation process, one or more processors 110 execute computer executable instructions stored in the machine-readable storage medium 120, so that the processor 110 can execute the remote warning method based on big data and image recognition in the above method embodiment. The processor 110, the machine-readable storage medium 120 and the communication unit 140 are connected through the bus 130, and the processor 110 can be used to control the sending and receiving actions of the communication unit 140.

[0108] The specific implementation process of the processor 110 can refer to the various method embodiments executed by the above-mentioned remote warning system 100 based on big data and image recognition. The implementation principles and technical effects are similar, and this embodiment will not be repeated here.

[0109] In addition, an embodiment of the present invention also provides a readable storage medium, in which computer executable instructions are preset. When the processor executes the computer executable instructions, the above-mentioned remote warning method based on big data and image recognition is implemented.

[0110] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, various features are sometimes combined into one embodiment, drawing or description thereof.

Claims

1. A remote early warning method based on big data and image recognition, characterized in that: The method comprises: Based on the template Internet of Vehicles driving data and the template vehicle-mounted image data, a fusion perception operation is performed through a fusion perception neural network to generate a first driving behavior description vector of the template Internet of Vehicles driving data and a first image semantic vector of the template vehicle-mounted image data; Performing feature space conversion on the first driving behavior description vector and the first image semantic vector respectively to generate a second driving behavior description vector and a second image semantic vector; determining a first comparison error according to the first driving behavior description vector and the second image semantic vector, and determining a second comparison error according to the second driving behavior description vector and the first image semantic vector; Determining a global comparison error corresponding to the fused perception neural network according to the first comparison error and the second comparison error; The fused perception neural network is optimized according to the global comparison error to generate the optimized fused perception neural network, so as to perform remote risk control warning on any input candidate Internet of Vehicles driving data and candidate vehicle-mounted image data based on the optimized fused perception neural network.

2. The remote early warning method based on big data and image recognition according to claim 1 is characterized in that: The determining of a first comparison error based on the first driving behavior description vector and the second image semantic vector, and the determining of a second comparison error based on the second driving behavior description vector and the first image semantic vector, include: Calculating the actual feature distance between the following feature information: the template vehicle driving state data corresponding to the first driving behavior description vector and the template vehicle image content corresponding to the first image semantic vector; Determining a first characteristic distance between the first driving behavior description vector and the second image semantic vector, and determining a second characteristic distance between the second driving behavior description vector and the first image semantic vector; A first cost coefficient between the actual feature distance and the first feature distance is used as the first comparison error, and a first cost coefficient between the actual feature distance and the second feature distance is used as the second comparison error.

3. The remote early warning method based on big data and image recognition according to claim 1 is characterized in that: Determining the global comparison error corresponding to the fused perception neural network according to the first comparison error and the second comparison error includes: Determine a first average error of each of the first comparison errors, and determine a second average error of each of the second comparison errors, and use a weighted result of the first average error and the second average error as a weighted comparison error of the fused perceptual neural network; Performing feature interaction on the first driving behavior description vector and the first image semantic vector in the template vector combination to generate a feature interaction result; Estimating based on the feature interaction result, generating a first confidence that the template vector combination is a valid template, wherein when the template vehicle driving state data corresponding to the first driving behavior description vector and the template vehicle image content corresponding to the first image semantic vector reflect the same risk control warning event, the template vector combination is a valid template; Determine the annotation confidence that the template vector combination is a valid template, determine a third cost coefficient between the first confidence and the annotation confidence of each template vector combination, and use the weighted result of the third cost coefficient as the combination error of the fusion perception neural network; The weighted comparison error and the combined error are fused to generate the global comparison error.

4. The remote early warning method based on big data and image recognition according to claim 3 is characterized in that: The step of performing feature interaction on the first driving behavior description vector and the first image semantic vector in the template vector combination to generate a feature interaction result includes: Acquire a first driving behavior description vector and a first image semantic vector generated by the fusion perception neural network, wherein the first driving behavior description vector includes a plurality of driving state fusion vectors, each driving state fusion vector is a comprehensive representation obtained after encoding and context information mapping of each driving state in the template vehicle driving state data, and the first image semantic vector is obtained by encoding and fusion of image feature points in the template vehicle-mounted image content; Standardizing the first driving behavior description vector and the first image semantic vector to generate a standardized first driving behavior description vector and a standardized first image semantic vector; Based on the standardized first driving behavior description vector and the standardized first image semantic vector, an association matrix is ​​constructed, and the association matrix is ​​used to represent the potential connection between driving behavior features and image semantic features; the construction process includes: calculating the similarity between each driving state fusion vector in the first driving behavior description vector and each image feature point code in the first image semantic vector, and using the calculated similarity as an element of the association matrix, wherein the rows of the association matrix correspond to the driving state fusion vectors, and the columns of the association matrix correspond to the image feature point codes, and generating an association matrix representing the association strength between driving behaviors and image contents; Using the association matrix, the driving state fusion vector in the first driving behavior description vector is mapped to the image content space, and the image content space mapping vector corresponding to each driving state fusion vector is generated; the specific operation is: for each driving state fusion vector, according to the similarity between the driving state fusion vector and each image feature point encoding in the association matrix, a weighted sum is calculated, wherein the weighted weight is the similarity value, and the result of the weighted sum is used as the mapping representation of the driving state in the image content space; The image feature point codes in the first image semantic vector are mapped to the driving behavior space to generate a driving behavior space mapping vector corresponding to each image feature point code. The specific operation is as follows: for each image feature point code, a weighted sum is calculated according to the similarity between the image feature point code and each driving state fusion vector in the association matrix to obtain a mapping representation of the image feature point in the driving behavior space; By using a neural network fusion algorithm, the image content space mapping vector of each driving state fusion vector is fused with the driving behavior space reflection vector encoded by each image feature point to generate a fused bidirectional mapping interaction vector group, wherein the fused bidirectional mapping interaction vector group contains a feature representation of the driving behavior and the image content that are mutually enhanced; The fused bidirectional mapping interaction vector group is input into one or more layers of neural network layers for nonlinear transformation and feature extraction, so as to generate a hierarchically abstracted and refined target interaction feature representation; The target interaction feature representations are integrated through a global average pooling method or a maximum pooling method to generate a final feature interaction result.

5. The remote early warning method based on big data and image recognition according to claim 1 is characterized in that: The method of performing a fusion perception operation based on the template Internet of Vehicles driving data and the template vehicle-mounted image data through a fusion perception neural network to generate a first driving behavior description vector of the template Internet of Vehicles driving data and a first image semantic vector of the template vehicle-mounted image data includes: By fusing the perception neural network, according to the first driving state coding bitmap, the driving state coding of each template vehicle driving state data in the template vehicle networking driving data is performed, and the first state coding result of each driving state in each template vehicle driving state data is generated, the template vehicle driving state data includes a plurality of driving states, and the first driving state coding bitmap includes a coding conversion relationship between the driving state and the first state coding result; Based on the context information of each driving state in the template vehicle driving state data, performing state node mapping on the first state encoding result of each driving state to generate a driving state fusion vector of each driving state; For each of the template vehicle driving state data, the driving state fusion vectors of the multiple driving states included in the template vehicle driving state data are fused to generate a first driving behavior description vector; According to the second driving state coding bitmap, each template vehicle-mounted image content in the template vehicle-mounted image data is coded for the driving state, and a second state coding result of each image feature point in each template vehicle-mounted image content is generated, wherein the second driving state coding bitmap includes a coding conversion relationship between the image feature point and the second state coding result; For each of the template vehicle-mounted image contents, the second state encoding result of each of the image feature points in the template vehicle-mounted image contents is fused to generate a first image semantic vector.

6. The remote early warning method based on big data and image recognition according to claim 5 is characterized in that: The step of performing state node mapping on the first state encoding result of each driving state based on the context information of each driving state in the template vehicle driving state data to generate a driving state fusion vector for each driving state comprises: For each driving state in the template vehicle driving state data, extracting context information corresponding to the driving state, the context information including the preceding and following adjacent states of the driving state in the time series, vehicle dynamic parameters associated with the driving state, environmental perception data, and a driving behavior mode identifier; Using the context information, enhancing the representation of the first state encoding result of each driving state to generate an enhanced driving state representation set; Construct a state node mapping network, the state node mapping network is used to map the enhanced driving state representation set to the driving state fusion vector space, the state node mapping network is a deep learning model, including a multilayer perceptron, a convolutional neural network or a graph neural network; the input of the state node mapping network is the enhanced driving state representation, and the output is the driving state fusion vector. During the training process of the state node mapping network, a set of labeled driving state data is used as a training set, wherein each driving state corresponds to a predefined driving state fusion vector, and the network parameters of the state node mapping network are optimized by minimizing the difference between the output of the state node mapping network and the target fusion vector, so that the state node mapping network maps the enhanced driving state representation to the driving state fusion vector space; The enhanced driving state representation set is mapped using the state node mapping network to generate a driving state fusion vector for each driving state.

7. The remote early warning method based on big data and image recognition according to claim 1 is characterized in that: The performing feature space conversion on the first driving behavior description vector and the first image semantic vector respectively to generate a second driving behavior description vector and a second image semantic vector includes: Editing each driving state fusion vector in the first driving behavior description vector into numerical information to generate the second driving behavior description vector; Perform dimensionality reduction on the second state encoding result corresponding to each image feature point in the first image semantic vector to generate the second image semantic vector.

8. The remote early warning method based on big data and image recognition according to any one of claims 1 to 7, characterized in that: The step of performing remote risk control warning on any input candidate Internet of Vehicles driving data and candidate vehicle-mounted image data based on the optimized fusion perception neural network includes: Acquire a training data sequence, the training data sequence comprising a plurality of training data combinations, and a labeling confidence corresponding to a template risk control warning event associated with each training data combination, the training data combination comprising: template vehicle driving state data and template vehicle-mounted image content of the template risk control warning event; According to the multiple training data combinations, the optimized fusion perception neural network is used to estimate and generate a second confidence level corresponding to the template risk control warning event associated with each training data combination; According to the cost coefficient between each of the second confidences and the corresponding labeled confidence, optimizing the neuron weight information of the fused perception neural network to generate the optimized fused perception neural network; The optimized fusion perception neural network is used to predict the candidate vehicle driving status data and candidate vehicle-mounted image content in the Internet of Vehicles big data of any input target user, generate an estimated confidence level of the candidate risk control warning event matched by the candidate vehicle driving status data, and determine the target risk control warning event matched by the candidate vehicle driving status data based on the estimated confidence level of the candidate risk control warning event matched by the candidate vehicle driving status data.

9. The remote early warning method based on big data and image recognition according to claim 8 is characterized in that: The method further comprises: According to each target risk control warning event of the target user, the product content of each insurance service product subscribed by the target user is optimized.

10. A remote early warning system based on big data and image recognition, characterized in that: The remote warning system based on big data and image recognition includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the remote warning method based on big data and image recognition as described in any one of claims 1 to 9.

Images