AR-Based Vehicle Speed Assistance System and Its Method

Abstract

This application relates to the field of intelligent warning. Specifically, it discloses an AR-based vehicle speed assistance system and method. By using a neural network model based on deep learning, it mines the complex mapping relationship between the relative temporal changes in the vehicle distance in the detected image of the vehicle in front and the changes in the vehicle speed value, so as to assist in warning whether a vehicle collision will occur for the current vehicle speed, thereby ensuring the driving safety of the driver.

Description

Technical Field

[0001] The present application relates to the field of intelligent warning, and more specifically, to an AR-based vehicle speed assistance system and method thereof. Background Art

[0002] All along, a driver's judgment of other vehicles has been determined based on the indicator lights of surrounding vehicles and the driver's own experience. Besides the potential safety hazards caused by the driver's inattention and subjective factors leading to judgment errors, the indicator lights of surrounding vehicles that do not give corresponding indications before taking actions are also an important safety hazard. For example, when the taillights of a surrounding vehicle are damaged, there will be no indicator light to prompt the following vehicle to pay attention to braking and decelerating when it brakes and decelerates.

[0003] AR enhanced display technology can obtain information about surrounding targets through Internet technology within the normal field of view of the human eye and display the information of surrounding targets on a specific display interface to increase the interaction of people within a specific range. The display interface is mainly a display device such as glasses that is close to the user's eyeball at present, and the other type is the display screen of a mobile phone. With the development of vehicle internal projection technology, there has already been a technology that projects vehicle own parameters such as the current speed, gear, etc. onto the vehicle front windshield. If the two technologies are integrated, it will open up a new application field for AR technology.

[0004] Therefore, there is an expectation for an AR-based vehicle speed assistance system. Summary of the Invention

[0005] To solve the above technical problems, the present application is proposed. Embodiments of the present application provide an AR-based vehicle speed assistance system and method thereof. By using a neural network model based on deep learning, it mines the complex mapping relationship between the relative temporal changes of the vehicle distance in the front vehicle detection image and the change of the vehicle speed value, so as to assist in warning whether a vehicle collision will occur for the current vehicle speed to ensure the driving safety of the driver.

[0006] According to one aspect of the present application, there is provided an AR-based vehicle speed assistance system, which includes:

[0007] A data acquisition module, configured to acquire the front vehicle detection images at multiple predetermined time points within a predetermined time period collected by a camera and the vehicle speed values at the multiple predetermined time points;

[0008] An image feature extraction module, configured to respectively pass the front vehicle detection images at the multiple predetermined time points through a convolutional neural network model as a filter to obtain multiple front vehicle detection feature vectors;

[0009] A timing correlation encoding module, configured to obtain a relative position timing feature vector of the preceding vehicle by using a transformer-based context encoder for the multiple preceding vehicle detection feature vectors;

[0010] A vehicle speed timing feature extraction module, configured to arrange the vehicle speed values at the multiple predetermined time points into a vehicle speed input vector according to the time dimension and then obtain a vehicle speed timing feature vector through a multi-scale neighborhood feature extraction module;

[0011] An association module, configured to perform association encoding on the vehicle speed timing feature vector and the relative position timing feature vector of the preceding vehicle to obtain a classification feature matrix;

[0012] An optimization module, configured to perform feature distribution modulation on the classification feature matrix to obtain an optimized classification feature matrix;

[0013] An early warning module, configured to obtain a classification result by passing the optimized classification feature matrix through a classifier, where the classification result is used to indicate whether a vehicle speed early warning prompt is generated; and

[0014] A display module, configured to display the vehicle speed early warning prompt on an in-vehicle screen in response to the classification result indicating that a vehicle speed early warning prompt is generated.

[0015] In the above AR-based vehicle speed assistance system, the image feature extraction module is configured to: use each layer of the convolutional neural network model used as a filter to respectively perform the following on the input data during the forward pass of the layer: perform convolutional processing on the input data to obtain a convolutional feature map; perform pooling based on a feature matrix on the convolutional feature map to obtain a pooled feature map; and perform non-linear activation on the pooled feature map to obtain an activated feature map; where the output of the last layer of the convolutional neural network used as a filter is the multiple preceding vehicle detection feature vectors, and the input of the first layer of the convolutional neural network used as a filter is the multiple preceding vehicle detection images at the multiple predetermined time points.

[0016] In the above AR-based vehicle speed assistance system, the timing correlation encoding module includes: a context encoding unit, configured to perform global context semantic encoding based on the idea of a transformer on the multiple preceding vehicle detection feature vectors by using the transformer of the context encoder including an embedding layer to obtain multiple global context semantic preceding vehicle detection feature vectors; and a concatenation unit, configured to concatenate the multiple global context semantic preceding vehicle detection feature vectors to obtain the relative position timing feature vector of the preceding vehicle.

[0017] In the above AR-based vehicle speed assistance system, the context encoding unit includes: a query vector construction sub-unit, configured to arrange the multiple preceding vehicle detection feature vectors in one dimension to obtain a global preceding vehicle detection feature vector; a self-attention sub-unit, configured to calculate the product between the global preceding vehicle detection feature vector and the transposed vectors of the respective preceding vehicle detection feature vectors among the multiple preceding vehicle detection feature vectors to obtain multiple self-attention correlation matrices; a normalization sub-unit, configured to perform normalization processing on each of the multiple self-attention correlation matrices to obtain multiple normalized self-attention correlation matrices; an attention degree calculation sub-unit, configured to pass each of the multiple normalized self-attention correlation matrices through a Softmax classification function to obtain multiple probability values; an attention application sub-unit, configured to respectively use each of the probability values among the multiple probability values as weights to weight the respective preceding vehicle detection feature vectors among the multiple preceding vehicle detection feature vectors to obtain the multiple context semantic preceding vehicle detection feature vectors; and a concatenation sub-unit, configured to concatenate the multiple context semantic preceding vehicle detection feature vectors to obtain the global context semantic preceding vehicle detection feature vector.

[0018] In the above AR-based vehicle speed assistance system, the multi-scale neighborhood feature extraction module includes: a first convolutional layer, a second convolutional layer parallel to the first convolutional layer, and a multi-scale feature fusion layer connected to the first convolutional layer and the second convolutional layer, wherein the first convolutional layer uses a one-dimensional convolutional kernel with a first length, and the second convolutional layer uses a one-dimensional convolutional kernel with a second length.

[0019] In the above AR-based vehicle speed assistance system, the vehicle speed time series feature extraction module includes: a first neighborhood scale feature extraction unit, configured to input the vehicle speed input vector into the first convolutional layer of the multi-scale neighborhood feature extraction module to obtain a first neighborhood scale vehicle speed time series feature vector, wherein the first convolutional layer has a first one-dimensional convolutional kernel with a first length; a second neighborhood scale feature extraction unit, configured to input the vehicle speed input vector into the second convolutional layer of the multi-scale neighborhood feature extraction module to obtain a second neighborhood scale vehicle speed time series feature vector, wherein the second convolutional layer has a second one-dimensional convolutional kernel with a second length, and the first length is different from the second length; and a multi-scale fusion unit, configured to concatenate the first neighborhood scale vehicle speed time series feature vector and the second neighborhood scale vehicle speed time series feature vector to obtain the vehicle speed time series feature vector. Wherein, the first neighborhood scale feature extraction unit is configured to: use the first convolutional layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolutional encoding on the vehicle speed input vector according to the following formula to obtain a first neighborhood scale vehicle speed time series feature vector; wherein, the formula is:

[0020]

[0021] Among them, a is the width of the first convolution kernel in the x direction, F(a) is the first convolution kernel parameter vector, G(x - a) is the local vector matrix for operating with the convolution kernel function, w is the size of the first convolution kernel, and X represents the vehicle speed input vector; and, the second neighborhood scale feature extraction unit is used for: using the second convolution layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolution encoding on the vehicle speed input vector according to the following formula to obtain the second neighborhood scale vehicle speed time series feature vector; where the formula is:

[0022]

[0023] Among them, b is the width of the second convolution kernel in the x direction, F(b) is the second convolution kernel parameter vector, G(x - b) is the local vector matrix for operating with the convolution kernel function, m is the size of the second convolution kernel, and X represents the vehicle speed input vector.

[0024] In the above AR-based vehicle speed assistance system, the association module is used for: performing association encoding on the vehicle speed time series feature vector and the relative position time series feature vector of the vehicle in front according to the following formula to obtain a classification feature matrix; where the formula is:

[0025]

[0026] Among them, V m represents the vehicle speed time series feature vector, represents the transposed vector of the vehicle speed time series feature vector, V n represents the relative position time series feature vector of the vehicle in front, M1 represents the classification feature matrix, represents vector multiplication.

[0027] In the above AR-based vehicle speed assistance system, the optimization module includes: an unfolding unit for unfolding the classification feature matrix row by row or column by column into a classification feature vector; a feature optimization unit for performing vector norm Hilbert probability space transformation on the classification feature vector according to the following formula to obtain an optimized classification feature vector; where the formula is:

[0028]

[0029] Among them, V is the classification feature vector, ‖V‖2 represents the second norm of the classification feature vector, represents the square of the second norm of the classification feature vector, v iis the i-th eigenvalue of the classification feature vector, exp(·) represents the exponential operation of the vector, and the exponential operation of the vector means calculating the natural exponential function values with the eigenvalues at each position in the vector as the exponents, and v i ′ is the i-th eigenvalue of the optimized classification feature vector; and, a matrix reconstruction unit, configured to perform matrix reconstruction on the optimized classification feature vector to obtain the optimized classification feature matrix.

[0030] In the above AR-based vehicle speed assistance system, the warning module includes: a classification feature vector generation unit, configured to expand the classification feature matrix into a classification feature vector based on row vectors or column vectors; a fully connected encoding unit, configured to perform fully connected encoding on the classification feature vector using multiple fully connected layers of the classifier to obtain an encoded classification feature vector; and, a classification result generation unit, configured to pass the encoded classification feature vector through the Softmax classification function of the classifier to obtain the classification result.

[0031] According to another aspect of the present application, there is provided an AR-based vehicle speed assistance method, which includes:

[0032] Obtain the front vehicle detection images at multiple predetermined time points within a predetermined time period collected by the camera and the vehicle speed values at the multiple predetermined time points;

[0033] Respectively pass the front vehicle detection images at the multiple predetermined time points through a convolutional neural network model as a filter to obtain multiple front vehicle detection feature vectors;

[0034] Pass the multiple front vehicle detection feature vectors through a context encoder based on a transformer to obtain a front vehicle relative position time series feature vector;

[0035] Arrange the vehicle speed values at the multiple predetermined time points in the time dimension into a vehicle speed input vector and then pass it through a multi-scale neighborhood feature extraction module to obtain a vehicle speed time series feature vector;

[0036] Perform correlation encoding on the vehicle speed time series feature vector and the front vehicle relative position time series feature vector to obtain a classification feature matrix;

[0037] Perform feature distribution modulation on the classification feature matrix to obtain an optimized classification feature matrix;

[0038] Pass the classification feature matrix through a classifier to obtain a classification result, and the classification result is used to indicate whether a vehicle speed warning prompt is generated; and

[0039] In response to the classification result being that a vehicle speed warning prompt is generated, display the vehicle speed warning prompt on the in-vehicle screen.

[0040] According to another aspect of the present application, there is provided an electronic device, including: a processor; and a memory in which computer program instructions are stored, and when the computer program instructions are run by the processor, the processor is caused to execute the AR-based vehicle speed assistance method as described above.

[0041] According to yet another aspect of the present application, there is provided a computer-readable medium having computer program instructions stored thereon, and when the computer program instructions are run by a processor, the processor is caused to execute the AR-based vehicle speed assistance method as described above.

[0042] Compared with the prior art, the AR-based vehicle speed assistance system and method provided by the present application adopt a neural network model based on deep learning to mine the complex mapping relationship between the relative temporal change of the vehicle distance in the front vehicle detection image and the change of the vehicle speed value, so as to assist in warning whether a vehicle collision will occur for the current vehicle speed, so as to ensure the driving safety of the driver. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] By describing the embodiments of the present application in more detail in conjunction with the drawings, the above and other objects, features and advantages of the present application will become more obvious. The drawings are used to provide a further understanding of the embodiments of the present application, and constitute a part of the specification, and are used to explain the present application together with the embodiments of the present application, and do not constitute a limitation to the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0044] Figure 1 FIG. is an application scenario diagram of the AR-based vehicle speed assistance system according to an embodiment of the present application;

[0045] Figure 2 FIG. is a block diagram of the AR-based vehicle speed assistance system according to an embodiment of the present application;

[0046] Figure 3 FIG. is a system architecture diagram of the AR-based vehicle speed assistance system according to an embodiment of the present application;

[0047] Figure 4 FIG. is a flowchart of convolutional neural network encoding in the AR-based vehicle speed assistance system according to an embodiment of the present application;

[0048] Figure 5 FIG. is a block diagram of a vehicle speed temporal feature extraction module in the AR-based vehicle speed assistance system according to an embodiment of the present application;

[0049] Figure 6 FIG. is a block diagram of an optimization module in the AR-based vehicle speed assistance system according to an embodiment of the present application;

[0050] Figure 7 Block diagram of the warning module in the AR-based vehicle speed assistance system according to an embodiment of the present application;

[0051] Figure 8 Flowchart of the AR-based vehicle speed assistance method according to an embodiment of the present application;

[0052] Figure 9 Block diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0053] Hereinafter, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments of the present application. It should be understood that the present application is not limited by the exemplary embodiments described herein.

[0054] Application overview

[0055] As described above, the AR enhanced display technology can obtain information of surrounding targets through the Internet technology within the normal visual field of the human eye, and display the information of the surrounding targets on a specific display interface, increasing the interaction of people within a specific range. The display interface is mainly a display device close to the user's eyeball such as glasses at present stage, and the other is the display screen of a mobile phone. With the development of vehicle interior projection technology, there has been a technology of projecting vehicle own parameters such as current speed, gear, etc. on the vehicle front windshield. If the two technologies are integrated, a new application field of AR technology will be opened up. Therefore, an AR-based vehicle speed assistance system is expected.

[0056] Correspondingly, when considering combining the AR enhanced display technology and the vehicle interior projection technology to assist in warning the vehicle speed, that is, when the vehicle speed is too fast, it may lead to an accident of vehicle collision. In order to avoid such accidents, it is necessary to comprehensively analyze information such as the vehicle speed and distance of the vehicle based on the AR enhanced display technology and the vehicle interior projection technology, and generate a warning prompt that the vehicle speed may cause a vehicle collision to ensure the driving safety of the driver. Specifically, it can be analyzed by collecting the change information of the front vehicle detection image and the vehicle speed value in the time dimension to warn the vehicle speed. Here, the relative position information of the two vehicles exists in the front vehicle detection image. However, due to the large amount of information in the image data, it is difficult to capture the effective information. In this process, the difficulty lies in how to establish the mapping relationship between the relative temporal change of the vehicle distance in the front vehicle detection image and the change of the vehicle speed value, so as to assist in warning whether a vehicle collision will occur for the current vehicle speed to ensure the driving safety of the driver.

[0057] In recent years, deep learning and neural networks have been widely applied in the fields of computer vision, natural language processing, text signal processing, etc. In addition, deep learning and neural networks have also demonstrated levels approaching or even exceeding those of humans in the fields of image classification, object detection, semantic segmentation, text translation, etc.

[0058] The development of deep learning and neural networks provides new solutions and ideas for mining the complex mapping relationship between the relative temporal changes in the vehicle distance in the preceding vehicle detection image and the changes in the vehicle speed value. Those of ordinary skill in the art should be aware that a deep neural network model based on deep learning can adjust the parameters of the deep neural network model through appropriate training strategies, such as the backpropagation algorithm of gradient descent, so that it can simulate the complex non-linear associations between things, and this is obviously suitable for simulating and establishing the complex mapping relationship between the relative temporal changes in the vehicle distance in the preceding vehicle detection image and the changes in the vehicle speed value.

[0059] Specifically, in the technical solution of this application, first, the preceding vehicle detection images at multiple predetermined time points within a predetermined time period are collected through a camera, and the vehicle speed values at the multiple predetermined time points are obtained. Then, considering that since the preceding vehicle detection image is image data and contains a large amount of information, it is difficult to effectively capture the relative position information of the two vehicles. Therefore, in the technical solution of this application, the convolutional neural network model as a filter, which has excellent performance in extracting implicit features of the image, is further used to respectively mine the features of the preceding vehicle detection images at each of the predetermined time points, so as to extract the implicit feature information of the relative position of the vehicle in the preceding vehicle detection image at each of the predetermined time points, that is, the distance feature information between the two vehicles, thereby obtaining a plurality of preceding vehicle detection feature vectors.

[0060] Then, considering that the relative position feature information of the vehicle has a dynamic law in the time dimension, that is, when the speed of one of the two vehicles decreases, the vehicle distance between the two vehicles will increase, and when the vehicle speeds of the two vehicles remain unchanged, the vehicle distance between the two vehicles will also remain unchanged. Therefore, the relative position feature of the vehicle has dynamic association feature information in time series. Based on this, in the technical solution of this application, the plurality of preceding vehicle detection feature vectors are encoded through a context encoder based on a transformer to extract the dynamic association feature information of the relative position feature of the vehicle in the preceding vehicle detection image in the time dimension, thereby obtaining a preceding vehicle relative position time series feature vector.

[0061] Further, for the vehicle speed values at the multiple predetermined time points, considering that the vehicle speed values are volatile and uncertain in the time dimension and have different pattern state change characteristic information under different time cycle spans within the predetermined time period. Therefore, in the technical solution of the present application, in order to fully extract the dynamic change characteristics of the vehicle speed values in time series, the vehicle speed values at the multiple predetermined time points are further arranged into a vehicle speed input vector according to the time dimension and then processed through a multi-scale neighborhood feature extraction module to extract the dynamic multi-scale neighborhood correlation features of the vehicle speed values under different time spans, thereby obtaining a vehicle speed time series feature vector.

[0062] Next, the vehicle speed time series feature vector and the relative position time series feature vector of the preceding vehicle can be associated and encoded to obtain a classification feature matrix. That is, calculate the vector multiplication of the vehicle speed time series feature vector and the relative position time series feature vector of the preceding vehicle to represent the correlation feature distribution information of the time series multi-scale dynamic features of the vehicle speed value and the relative position time series correlation features of the vehicle, and use this as the classification feature matrix. Then, pass the classification feature matrix through a classifier to obtain a classification result indicating whether a vehicle speed warning prompt is generated. That is, in the technical solution of the present application, the labels of the classifier include generating a vehicle speed warning prompt and not generating a vehicle speed warning prompt. Among them, the classifier determines which classification label the classification feature matrix belongs to through a softmax function to warn of potential vehicle collisions when the vehicle speed is too fast. And in response to the classification result being that a vehicle speed warning prompt is generated, display the vehicle speed warning prompt on the in-vehicle screen to ensure the driving safety of the driver.

[0063] Particularly, in the technical solution of the present application, when the vehicle speed time series feature vector and the relative position time series feature vector of the preceding vehicle are associated and encoded to obtain a classification feature matrix, since the relative position time series feature vector of the preceding vehicle expresses the high-order correlation features of the cross-time-domain correlation of the preceding vehicle image features (essentially representing the high-order correlation features of the spatial position relationship between the preceding vehicle and the vehicle in the time dimension), and the vehicle speed time series feature vector expresses the time series correlation features of the vehicle speed data of the vehicle itself, the classification feature matrix can fuse heterogeneous correlation features of different orders, thereby improving its feature expression ability. However, on the other hand, the superposition of features in different orders and source domains may also cause the overall feature distribution of the classification feature matrix as an inter-domain fusion feature to be relatively discrete, resulting in poor dependence of the classification feature matrix on a single classification result during classification, affecting the accuracy of the classification result.

[0064] Therefore, preferably, the classification feature vector obtained by expanding the classification feature matrix is subjected to vector norm Hilbert probability space transformation, specifically expressed as:

[0065]

[0066] V is the classification feature vector, and ‖V‖2 represents the second norm of the classification feature vector, representing its square, i.e., the inner product of the classification feature vector itself, v i is the i-th eigenvalue of the classification feature vector V, and v i ′ is the i-th eigenvalue of the optimized classification feature vector V′. Here, the vector norm-based Hilbert probability space transformation performs a probabilistic interpretation of the classification feature vector V within the Hilbert space defined by the norm of the classification feature vector V itself, and reduces the hidden perturbation of the class expression of each local distribution of the splicing of the classification feature vector V to the class expression of the overall Hilbert space topology, thereby improving the robustness of the feature distribution of the classification feature vector V converging to a single predetermined classification result. At the same time, relying on the establishment of the metric-induced probability space structure to enhance the long-range dependence of the feature distribution of the classification feature vector V on a single classification result across classifiers. In this way, the dependence of the optimized classification feature vector V′ on a single classification result during classification by the classifier is enhanced, and the accuracy of the classification result is improved. In this way, it is possible to assist in warning in real time and accurately whether a vehicle collision will occur for the current vehicle speed to ensure the driving safety of the driver.

[0067] Based on this, the present application proposes an AR-based vehicle speed assistance system, which includes: a data acquisition module for acquiring the front vehicle detection images at multiple predetermined time points within a predetermined time period collected by a camera and the vehicle speed values at the multiple predetermined time points; an image feature extraction module for respectively passing the front vehicle detection images at the multiple predetermined time points through a convolutional neural network model as a filter to obtain multiple front vehicle detection feature vectors; a temporal correlation encoding module for passing the multiple front vehicle detection feature vectors through a context encoder based on a transformer to obtain a front vehicle relative position temporal feature vector; a vehicle speed temporal feature extraction module for arranging the vehicle speed values at the multiple predetermined time points in the time dimension as a vehicle speed input vector and then passing it through a multi-scale neighborhood feature extraction module to obtain a vehicle speed temporal feature vector; an association module for performing association encoding on the vehicle speed temporal feature vector and the front vehicle relative position temporal feature vector to obtain a classification feature matrix; an optimization module for modulating the feature distribution of the classification feature matrix to obtain an optimized classification feature matrix; a warning module for passing the optimized classification feature matrix through a classifier to obtain a classification result, where the classification result is used to indicate whether a vehicle speed warning prompt is generated; and a display module for, in response to the classification result being that a vehicle speed warning prompt is generated, displaying the vehicle speed warning prompt on an in-vehicle screen.

[0068] Figure 1 This is an application scenario diagram of an AR-based vehicle speed assistance system according to an embodiment of the present application. As Figure 1 shown, in this application scenario, multiple front vehicle detection images at multiple predetermined time points within a predetermined time period are obtained through a camera (for example, C as Figure 1 illustrated), and vehicle speed values at the multiple predetermined time points are obtained through a speed sensor (for example, S1 as Figure 1 illustrated). Then, the above information is input into a server (for example, S2 in Figure 1 ) where an AR-based vehicle speed assistance algorithm is deployed. The server can process the input information with the AR-based vehicle speed assistance algorithm to generate a classification result indicating whether a vehicle speed warning prompt is generated. In response to the classification result being that a vehicle speed warning prompt is generated, the vehicle speed warning prompt is displayed on the in-vehicle screen.

[0069] After introducing the basic principle of the present application, various non-limiting embodiments of the present application will be specifically introduced with reference to the accompanying drawings.

[0070] Exemplary System

[0071] Figure 2 This is a block diagram of an AR-based vehicle speed assistance system according to an embodiment of the present application. As Figure 2 shown, the AR-based vehicle speed assistance system 300 according to an embodiment of the present application includes: a data acquisition module 310; an image feature extraction module 320; a temporal correlation encoding module 330; a vehicle speed temporal feature extraction module 340; an association module 350; an optimization module 360; a warning module 370; and a display module 380.

[0072] Among them, the data acquisition module 310 is configured to obtain the front vehicle detection images at multiple predetermined time points within a predetermined time period collected by the camera and the vehicle speed values at the multiple predetermined time points; the image feature extraction module 320 is configured to respectively pass the front vehicle detection images at the multiple predetermined time points through a convolutional neural network model serving as a filter to obtain multiple front vehicle detection feature vectors; the temporal correlation encoding module 330 is configured to pass the multiple front vehicle detection feature vectors through a context encoder based on a transformer to obtain a front vehicle relative position temporal feature vector; the vehicle speed temporal feature extraction module 340 is configured to arrange the vehicle speed values at the multiple predetermined time points into a vehicle speed input vector in the time dimension and then pass it through a multi-scale neighborhood feature extraction module to obtain a vehicle speed temporal feature vector; the association module 350 is configured to perform association encoding on the vehicle speed temporal feature vector and the front vehicle relative position temporal feature vector to obtain a classification feature matrix; the optimization module 360 is configured to perform feature distribution modulation on the classification feature matrix to obtain an optimized classification feature matrix; the warning module 370 is configured to pass the optimized classification feature matrix through a classifier to obtain a classification result, and the classification result is used to indicate whether a vehicle speed warning prompt is generated; and the display module 380 is configured to, in response to the classification result indicating that a vehicle speed warning prompt is generated, display the vehicle speed warning prompt on the in-vehicle screen.

[0073] Figure 3 is a system architecture diagram of the AR-based vehicle speed assistance system according to an embodiment of the present application. Combining Figure 2 and Figure 3, in this network architecture, first, the data acquisition module 310 acquires the front vehicle detection images at multiple predetermined time points within a predetermined time period collected by the camera and the vehicle speed values at the multiple predetermined time points; then, the image feature extraction module 320 respectively passes the front vehicle detection images at the multiple predetermined time points acquired by the data acquisition module 310 through a convolutional neural network model as a filter to obtain multiple front vehicle detection feature vectors; the temporal correlation encoding module 330 passes the multiple front vehicle detection feature vectors obtained by the image feature extraction module 320 through a context encoder based on a transformer to obtain a front vehicle relative position temporal feature vector; then, the vehicle speed temporal feature extraction module 340 arranges the vehicle speed values at the multiple predetermined time points acquired by the data acquisition module 310 in the time dimension into a vehicle speed input vector and then passes it through a multi-scale neighborhood feature extraction module to obtain a vehicle speed temporal feature vector; the association module 350 performs association encoding on the vehicle speed temporal feature vector obtained by the vehicle speed temporal feature extraction module 340 and the front vehicle relative position temporal feature vector obtained by the temporal correlation encoding module 330 to obtain a classification feature matrix; the optimization module 360 performs feature distribution modulation on the classification feature matrix calculated by the association module 350 to obtain an optimized classification feature matrix; the warning module 370 passes the optimized classification feature matrix obtained by the optimization module 360 through a classifier to obtain a classification result, and the classification result is used to indicate whether a vehicle speed warning prompt is generated; furthermore, the display module 380, in response to the classification result being that a vehicle speed warning prompt is generated, displays the vehicle speed warning prompt on the in-vehicle screen.

[0074] Specifically, during the operation of the AR-based vehicle speed assistance system 300, the data acquisition module 310 is used to acquire the front vehicle detection images at multiple predetermined time points within a predetermined time period collected by the camera and the vehicle speed values at the multiple predetermined time points. It should be understood that during the process of generating a warning prompt for vehicle driving collision, the vehicle speed, distance and other information of the vehicle can be comprehensively analyzed by combining the AR enhanced display technology and the vehicle interior projection technology to generate a warning prompt that the vehicle speed may cause a vehicle collision. Specifically, in a specific example of the present application, first, the front vehicle detection images at multiple predetermined time points within a predetermined time period can be acquired through a camera, and the vehicle speed values at the multiple predetermined time points can be acquired through a speed sensor. Further, based on the mapping relationship between the relative temporal change of the vehicle distance in the front vehicle detection image and the change of the vehicle speed value, the auxiliary warning for whether a vehicle collision will occur at the current vehicle speed is carried out to ensure the driving safety of the driver.

[0075] Specifically, during the operation of the AR-based vehicle speed assistance system 300, the image feature extraction module 320 is configured to respectively pass the front vehicle detection images at the multiple predetermined time points through a convolutional neural network model as a filter to obtain multiple front vehicle detection feature vectors. In the technical solution of the present application, since the front vehicle detection images are image data and contain a large amount of information, it is difficult to effectively capture the relative position information of the two vehicles. Therefore, in the technical solution of the present application, a convolutional neural network model as a filter, which has excellent performance in implicit feature extraction of images, is further used to respectively mine the features of the front vehicle detection images at each of the predetermined time points, so as to extract the implicit feature information of the relative position of the vehicle in the front vehicle detection images at each of the predetermined time points, that is, the distance feature information between the two vehicles, thereby obtaining multiple front vehicle detection feature vectors. In a specific example, the convolutional neural network includes multiple neural network layers connected in cascade, and each neural network layer includes a convolutional layer, a pooling layer, and an activation layer. Among them, during the encoding process of the convolutional neural network, each layer of the convolutional neural network performs convolutional processing based on a convolution kernel on the input data, performs pooling processing on the convolutional feature map output by the convolutional layer using the pooling layer, and performs activation processing on the pooling feature map output by the pooling layer using the activation layer during the forward pass of the layer. Among them, the input of the first layer of the convolutional neural network is the front vehicle detection images at the multiple predetermined time points, and the output of the last layer of the convolutional neural network is the multiple front vehicle detection feature vectors.

[0076] Figure 4 FIG. is a flowchart of convolutional neural network encoding in an AR-based vehicle speed assistance system according to an embodiment of the present application. As Figure 4 shown, during the encoding process of the convolutional neural network, it includes: using each layer of the convolutional neural network model as a filter to respectively perform on the input data during the forward pass of the layer: S210, performing convolutional processing on the input data to obtain a convolutional feature map; S220, performing pooling based on a feature matrix on the convolutional feature map to obtain a pooling feature map; and S230, performing non-linear activation on the pooling feature map to obtain an activation feature map; where the output of the last layer of the convolutional neural network as a filter is the multiple front vehicle detection feature vectors, and the input of the first layer of the convolutional neural network as a filter is the front vehicle detection images at the multiple predetermined time points.

[0077] Specifically, during the operation of the AR-based vehicle speed assistance system 300, the timing correlation encoding module 330 is configured to obtain the relative position timing feature vector of the preceding vehicle by passing the multiple preceding vehicle detection feature vectors through a context encoder based on a transformer. In the technical solution of the present application, considering that the relative position feature information of the vehicle has a dynamic law in the time dimension, that is, when the speed of one of the two vehicles decreases, the distance between the two vehicles will increase, and when the vehicle speeds of the two vehicles remain unchanged, the distance between the two vehicles will also remain unchanged. Therefore, the relative position feature of the vehicle has timing dynamic correlation feature information. Based on this, in the technical solution of the present application, the multiple preceding vehicle detection feature vectors are encoded through a context encoder based on a transformer to extract the dynamic correlation feature information of the relative position of the vehicle in the time dimension in the preceding vehicle detection image, so as to obtain the relative position timing feature vector of the preceding vehicle. That is, based on the transformer idea, using the characteristic that the transformer can capture long-distance context dependencies, the multiple preceding vehicle detection feature vectors are subjected to global context semantic encoding to obtain a context semantic correlation feature representation with the overall semantic correlation of the multiple preceding vehicle detection feature vectors as the context background, that is, the global context semantic preceding vehicle detection feature vector. More specifically, to obtain the relative position timing feature vector of the preceding vehicle by passing the multiple preceding vehicle detection feature vectors through a context encoder based on a transformer, first, the transformer of the context encoder including an embedding layer is used to perform global context semantic encoding on the multiple preceding vehicle detection feature vectors based on the transformer idea to obtain multiple global context semantic preceding vehicle detection feature vectors, and then the multiple global context semantic preceding vehicle detection feature vectors are concatenated to obtain the relative position timing feature vector of the preceding vehicle.Among them, using the transformer of the context encoder including the embedding layer to perform global context semantic encoding on the multiple front vehicle detection feature vectors based on the transformer idea to obtain multiple global context semantic front vehicle detection feature vectors, including: arranging the multiple front vehicle detection feature vectors in a one-dimensional manner to obtain a global front vehicle detection feature vector; calculating the product between the global front vehicle detection feature vector and the transposed vectors of each front vehicle detection feature vector among the multiple front vehicle detection feature vectors to obtain multiple self-attention correlation matrices; respectively performing normalization processing on each self-attention correlation matrix among the multiple self-attention correlation matrices to obtain multiple normalized self-attention correlation matrices; passing each normalized self-attention correlation matrix among the multiple normalized self-attention correlation matrices through the Softmax classification function to obtain multiple probability values; respectively using each probability value among the multiple probability values as a weight to weight each front vehicle detection feature vector among the multiple front vehicle detection feature vectors to obtain the multiple context semantic front vehicle detection feature vectors; and concatenating the multiple context semantic front vehicle detection feature vectors to obtain the global context semantic front vehicle detection feature vector.

[0078] Specifically, during the operation of the AR-based vehicle speed assistance system 300, the vehicle speed time series feature extraction module 340 is configured to arrange the vehicle speed values at the multiple predetermined time points in a time dimension as a vehicle speed input vector and then obtain a vehicle speed time series feature vector through the multi-scale neighborhood feature extraction module. Considering that the vehicle speed values have volatility and uncertainty in the time dimension and have different pattern state change feature information under different time cycle spans within the predetermined time period. Therefore, in the technical solution of the present application, in order to fully extract the dynamic change features of the vehicle speed values in the time series, the vehicle speed values at the multiple predetermined time points are further arranged in a time dimension as a vehicle speed input vector and then processed through the multi-scale neighborhood feature extraction module to extract the dynamic multi-scale neighborhood correlation features of the vehicle speed values under different time spans, thereby obtaining a vehicle speed time series feature vector. Among them, the multi-scale neighborhood feature extraction module includes: a first convolutional layer, a second convolutional layer parallel to the first convolutional layer, and a multi-scale feature fusion layer connected to the first convolutional layer and the second convolutional layer, where the first convolutional layer uses a one-dimensional convolutional kernel with a first length, and the second convolutional layer uses a one-dimensional convolutional kernel with a second length.

[0079] Figure 5 It is a block diagram of the vehicle speed time series feature extraction module in the AR-based vehicle speed assistance system according to an embodiment of the present application. As Figure 5As shown, the vehicle speed time series feature extraction module 340 includes: a first neighborhood scale feature extraction unit 341, configured to input the vehicle speed input vector into the first convolutional layer of the multi-scale neighborhood feature extraction module to obtain a first neighborhood scale vehicle speed time series feature vector, where the first convolutional layer has a first one-dimensional convolutional kernel with a first length; a second neighborhood scale feature extraction unit 342, configured to input the vehicle speed input vector into the second convolutional layer of the multi-scale neighborhood feature extraction module to obtain a second neighborhood scale vehicle speed time series feature vector, where the second convolutional layer has a second one-dimensional convolutional kernel with a second length, and the first length is different from the second length; and a multi-scale fusion unit 343, configured to concatenate the first neighborhood scale vehicle speed time series feature vector and the second neighborhood scale vehicle speed time series feature vector to obtain the vehicle speed time series feature vector. Wherein, the first neighborhood scale feature extraction unit is configured to: use the first convolutional layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolutional encoding on the vehicle speed input vector according to the following formula to obtain a first neighborhood scale vehicle speed time series feature vector; where the formula is:

[0080]

[0081] Where a is the width of the first convolutional kernel in the x direction, F(a) is the first convolutional kernel parameter vector, G(x - a) is the local vector matrix for operation with the convolutional kernel function, w is the size of the first convolutional kernel, and X represents the vehicle speed input vector; and the second neighborhood scale feature extraction unit is configured to: use the second convolutional layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolutional encoding on the vehicle speed input vector according to the following formula to obtain the second neighborhood scale vehicle speed time series feature vector; where the formula is:

[0082]

[0083] Where b is the width of the second convolutional kernel in the x direction, F(b) is the second convolutional kernel parameter vector, G(x - b) is the local vector matrix for operation with the convolutional kernel function, m is the size of the second convolutional kernel, and X represents the vehicle speed input vector. More specifically, the multi-scale fusion unit is configured to: fuse the first neighborhood scale vehicle speed time series feature vector and the second neighborhood scale vehicle speed time series feature vector according to the following formula to obtain the vehicle speed time series feature vector; where the formula is:

[0084] V m = Concat[V1, V2]

[0085] Where V1 represents the first neighborhood scale vehicle speed time series feature vector, V2 represents the second neighborhood scale vehicle speed time series feature vector, Concat[·, ·] represents the concatenation function, and V mRepresents the vehicle speed time - series feature vector.

[0086] Specifically, during the operation of the AR - based vehicle speed assistance system 300, the association module 350 is used to perform association encoding on the vehicle speed time - series feature vector and the relative position time - series feature vector of the vehicle in front to obtain a classification feature matrix. That is, perform association encoding on the vehicle speed time - series feature vector and the relative position time - series feature vector of the vehicle in front to obtain a classification feature matrix. That is, calculate the vector multiplication of the vehicle speed time - series feature vector and the relative position time - series feature vector of the vehicle in front to represent the correlation feature distribution information of the time - series multi - scale dynamic feature of the vehicle speed value and the relative position time - series correlation feature of the vehicle. In a specific example of the present application, the vehicle speed time - series feature vector and the relative position time - series feature vector of the vehicle in front are associated and encoded with the following formula to obtain a classification feature matrix; where the formula is:

[0087]

[0088] Where V m Represents the vehicle speed time - series feature vector, Represents the transposed vector of the vehicle speed time - series feature vector, V n Represents the relative position time - series feature vector of the vehicle in front, and M1 represents the classification feature matrix, Represents vector multiplication.

[0089] Specifically, during the operation of the AR - based vehicle speed assistance system 300, the optimization module 360 is used to perform feature distribution modulation on the classification feature matrix to obtain an optimized classification feature matrix. In the technical solution of the present application, when performing association encoding on the vehicle speed time - series feature vector and the relative position time - series feature vector of the vehicle in front to obtain a classification feature matrix, since the relative position time - series feature vector of the vehicle in front expresses the high - order correlation feature of the cross - time - domain correlation of the vehicle image features in front (essentially representing the high - order correlation feature of the spatial position relationship between the vehicle in front and the vehicle itself in the time dimension), and the vehicle speed time - series feature vector expresses the time - series correlation feature of the vehicle speed data of the vehicle itself, the classification feature matrix can fuse heterogeneous correlation features of different orders, thereby improving its feature expression ability. However, on the other hand, the superposition of features in different orders and source domains may cause the overall feature distribution of the classification feature matrix as an inter - domain fusion feature to be relatively discrete, resulting in poor dependence of the classification feature matrix on a single classification result during classification and affecting the accuracy of the classification result. Therefore, preferably, the classification feature vector obtained by expanding the classification feature matrix is subjected to vector - normed Hilbert probability space transformation, specifically expressed as:

[0090]

[0091] where \(V\) is the classification feature vector, \(\|V\|_2\) represents the second norm of the classification feature vector, which represents its square, that is, the inner product of the classification feature vector with itself, \(v\) i is the \(i\)-th eigenvalue of the classification feature vector \(V\), and \(v\) i ' is the \(i\)-th eigenvalue of the optimized classification feature vector \(V'\). Here, the normed Hilbert probability space transformation of the vector is performed by norming the classification feature vector \(V\) itself within the Hilbert space defined with vector inner product to perform the probabilistic interpretation of the classification feature vector \(V\), and reduce the hidden perturbation of the class expression of each local distribution of the splicing of the classification feature vector \(V\) to the class expression of the overall Hilbert space topology, thereby improving the robustness of the convergence of the feature distribution of the classification feature vector \(V\) to a single predetermined classification result. At the same time, relying on the establishment of the metric-induced probability space structure to enhance the long-range dependence of the feature distribution of the classification feature vector \(V\) on a single classification result across classifiers. In this way, the dependence of the optimized classification feature vector \(V'\) on a single classification result during classification by the classifier is enhanced, and the accuracy of the classification result is improved. In this way, it is possible to assist in warning in real time and accurately whether a vehicle collision will occur for the current vehicle speed to ensure the driving safety of the driver.

[0092] Figure 6 is a block diagram of an optimization module in the AR-based vehicle speed assistance system according to an embodiment of the present application. As Figure 6 shown, the optimization module 360 includes: an expansion unit 361 for expanding the classification feature matrix row by row or column by column into a classification feature vector; a feature optimization unit 362 for performing normed Hilbert probability space transformation on the classification feature vector according to the following formula to obtain an optimized classification feature vector; where the formula is:

[0093]

[0094] where \(V\) is the classification feature vector, \(\|V\|_2\) represents the second norm of the classification feature vector, represents the square of the second norm of the classification feature vector, \(v\) i is the \(i\)-th eigenvalue of the classification feature vector, \(\exp(\cdot)\) represents the exponential operation of the vector, and the exponential operation of the vector represents calculating the natural exponential function values with the eigenvalues at each position in the vector as exponents, and \(v\) i ' is the \(i\)-th eigenvalue of the optimized classification feature vector; and a matrix reconstruction unit 363 for performing matrix reconstruction on the optimized classification feature vector to obtain the optimized classification feature matrix.

[0095] Specifically, during the operation of the AR-based vehicle speed assistance system 300, the warning module 370 and the display module 380 are used to obtain a classification result by passing the optimized classification feature matrix through a classifier. The classification result is used to indicate whether a vehicle speed warning prompt is generated, and in response to the classification result indicating the generation of a vehicle speed warning prompt, the vehicle speed warning prompt is displayed on the in-vehicle screen. That is, the optimized classification matrix is passed through the classifier to obtain a classification result indicating whether a vehicle speed warning prompt is generated. In a specific example of the present application, the classifier is used to process the optimized classification feature matrix according to the following formula to obtain a classification result, where the formula is:

[0096] O = softmax{(W n , B n ):…:(W1, B1)|Project(F)), where Project(F) represents projecting the optimized classification feature matrix into a vector, and W1 to W n are the weight matrices of each fully connected layer, and B1 to B n represent the bias vectors of each fully connected layer. Specifically, the classifier includes a plurality of fully connected layers and a Softmax layer cascaded with the last fully connected layer of the plurality of fully connected layers. Among them, in the classification process of the classifier, first, the optimized classification feature matrix is projected into a vector. For example, in a specific example, the optimized classification feature matrix is expanded into a classification feature vector along the row vector or column vector; then, the classification feature vector is subjected to multiple fully connected encodings using the plurality of fully connected layers of the classifier to obtain an encoded classification feature vector; further, the encoded classification feature vector is input into the Softmax layer of the classifier, that is, the Softmax classification function is used to classify the encoded classification feature vector to obtain a classification label. In the technical solution of the present application, the labels of the classifier include generating a vehicle speed warning prompt and not generating a vehicle speed warning prompt. Among them, the classifier determines which classification label the classification feature matrix belongs to through the softmax function, so as to warn of possible vehicle collisions when the vehicle speed is too fast, and in response to the classification result indicating the generation of a vehicle speed warning prompt, the vehicle speed warning prompt is displayed on the in-vehicle screen to ensure the driving safety of the driver.

[0097] Figure 7 is a block diagram of the warning module in the AR-based vehicle speed assistance system according to an embodiment of the present application. As Figure 7As shown, the early warning module 370 includes: a classification feature vector generation unit 371 for expanding the classification feature matrix into a classification feature vector based on a row vector or a column vector; a fully connected encoding unit 372 for using multiple fully connected layers of the classifier to fully connect encode the classification feature vector to obtain an encoded classification feature vector; and a classification result generation unit 373 for passing the encoded classification feature vector through the Softmax classification function of the classifier to obtain the classification result.

[0098] In summary, according to the embodiment of the present application, the AR-based vehicle speed assistance system 300 is explained, which uses a deep learning-based neural network model to mine the complex mapping relationship between the relative temporal changes in the vehicle distance in the front vehicle detection image and the changes in the vehicle speed value, so as to provide auxiliary warning for whether a vehicle collision will occur at the current vehicle speed, so as to ensure the driver's driving safety.

[0099] As described above, the AR-based vehicle speed assistance system according to the embodiment of the present application can be implemented in various terminal devices. In one example, the AR-based vehicle speed assistance system 300 according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the AR-based vehicle speed assistance system 300 can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the AR-based vehicle speed assistance system 300 can also be one of the many hardware modules of the terminal device.

[0100] Alternatively, in another example, the AR-based vehicle speed assistance system 300 and the terminal device may also be separate devices, and the AR-based vehicle speed assistance system 300 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in an agreed data format.

[0101] Exemplary Methods

[0102] Figure 8 FIG. 1 is a flow chart of a vehicle speed assistance method based on AR according to an embodiment of the present application. Figure 8As shown, the AR-based vehicle speed assistance method according to an embodiment of the present application includes the steps of: S110, obtaining the front vehicle detection images at multiple predetermined time points within a predetermined time period collected by a camera and the vehicle speed values at the multiple predetermined time points; S120, respectively passing the front vehicle detection images at the multiple predetermined time points through a convolutional neural network model as a filter to obtain multiple front vehicle detection feature vectors; S130, passing the multiple front vehicle detection feature vectors through a context encoder based on a transformer to obtain a front vehicle relative position time series feature vector; S140, arranging the vehicle speed values at the multiple predetermined time points in a time dimension to form a vehicle speed input vector and then passing it through a multi-scale neighborhood feature extraction module to obtain a vehicle speed time series feature vector; S150, performing correlation encoding on the vehicle speed time series feature vector and the front vehicle relative position time series feature vector to obtain a classification feature matrix; S160, performing feature distribution modulation on the classification feature matrix to obtain an optimized classification feature matrix; S170, passing the classification feature matrix through a classifier to obtain a classification result, where the classification result is used to indicate whether a vehicle speed warning prompt is generated; and S180, in response to the classification result indicating that a vehicle speed warning prompt is generated, displaying the vehicle speed warning prompt on an in-vehicle screen.

[0103] In one example, in the above AR-based vehicle speed assistance method, step S120 includes: using each layer of the convolutional neural network model as a filter to respectively perform the following operations on the input data during the forward pass of the layer: performing convolutional processing on the input data to obtain a convolutional feature map; performing pooling based on a feature matrix on the convolutional feature map to obtain a pooled feature map; and performing non-linear activation on the pooled feature map to obtain an activated feature map; where the output of the last layer of the convolutional neural network as a filter is the multiple front vehicle detection feature vectors, and the input of the first layer of the convolutional neural network as a filter is the front vehicle detection images at the multiple predetermined time points.

[0104] In one example, in the above AR-based vehicle speed assistance method, the step S130 includes: using a transformer of the context encoder including an embedding layer to perform global context semantic encoding on the multiple front vehicle detection feature vectors based on the idea of a transformer to obtain multiple global context semantic front vehicle detection feature vectors; and concatenating the multiple global context semantic front vehicle detection feature vectors to obtain the front vehicle relative position time series feature vector. Among them, using the transformer of the context encoder including an embedding layer to perform global context semantic encoding on the multiple front vehicle detection feature vectors to obtain multiple global context semantic front vehicle detection feature vectors includes: arranging the multiple front vehicle detection feature vectors in a one-dimensional manner to obtain a global front vehicle detection feature vector; calculating the product between the global front vehicle detection feature vector and the transposed vectors of the respective front vehicle detection feature vectors among the multiple front vehicle detection feature vectors to obtain multiple self-attention correlation matrices; respectively performing normalization processing on the respective self-attention correlation matrices among the multiple self-attention correlation matrices to obtain multiple normalized self-attention correlation matrices; passing the respective normalized self-attention correlation matrices among the multiple normalized self-attention correlation matrices through a Softmax classification function to obtain multiple probability values; respectively using the respective probability values among the multiple probability values as weights to weight the respective front vehicle detection feature vectors among the multiple front vehicle detection feature vectors to obtain the multiple context semantic front vehicle detection feature vectors; and concatenating the multiple context semantic front vehicle detection feature vectors to obtain the global context semantic front vehicle detection feature vector.

[0105] In one example, in the above AR-based vehicle speed assistance method, step S140 includes: inputting the vehicle speed input vector into the first convolutional layer of the multi-scale neighborhood feature extraction module to obtain a first neighborhood scale vehicle speed time series feature vector, where the first convolutional layer has a first one-dimensional convolutional kernel with a first length; inputting the vehicle speed input vector into the second convolutional layer of the multi-scale neighborhood feature extraction module to obtain a second neighborhood scale vehicle speed time series feature vector, where the second convolutional layer has a second one-dimensional convolutional kernel with a second length, and the first length is different from the second length; and concatenating the first neighborhood scale vehicle speed time series feature vector and the second neighborhood scale vehicle speed time series feature vector to obtain the vehicle speed time series feature vector. The multi-scale neighborhood feature extraction module includes: a first convolutional layer, a second convolutional layer parallel to the first convolutional layer, and a multi-scale feature fusion layer connected to the first convolutional layer and the second convolutional layer, where the first convolutional layer uses a one-dimensional convolutional kernel with a first length, and the second convolutional layer uses a one-dimensional convolutional kernel with a second length. Specifically, inputting the vehicle speed input vector into the first convolutional layer of the multi-scale neighborhood feature extraction module to obtain a first neighborhood scale vehicle speed time series feature vector includes: using the first convolutional layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolutional encoding on the vehicle speed input vector according to the following formula to obtain a first neighborhood scale vehicle speed time series feature vector; where the formula is:

[0106]

[0107] where a is the width of the first convolutional kernel in the x direction, F(a) is the first convolutional kernel parameter vector, G(x - a) is the local vector matrix for the operation with the convolutional kernel function, w is the size of the first convolutional kernel, and X represents the vehicle speed input vector; and inputting the vehicle speed input vector into the second convolutional layer of the multi-scale neighborhood feature extraction module to obtain a second neighborhood scale vehicle speed time series feature vector includes: using the second convolutional layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolutional encoding on the vehicle speed input vector according to the following formula to obtain the second neighborhood scale vehicle speed time series feature vector; where the formula is:

[0108]

[0109] where b is the width of the second convolutional kernel in the x direction, F(b) is the second convolutional kernel parameter vector, G(x - b) is the local vector matrix for the operation with the convolutional kernel function, m is the size of the second convolutional kernel, and X represents the vehicle speed input vector.

[0110] In one example, in the above AR-based vehicle speed assistance method, step S150 includes: performing correlation encoding on the vehicle speed time series feature vector and the relative position time series feature vector of the vehicle in front to obtain a classification feature matrix; where the formula is:

[0111]

[0112] Where V m represents the vehicle speed time series feature vector, represents the transposed vector of the vehicle speed time series feature vector, V n represents the relative position time series feature vector of the vehicle in front, M1 represents the classification feature matrix, represents vector multiplication.

[0113] In one example, in the above AR-based vehicle speed assistance method, step S160 includes: expanding the classification feature matrix row by row or column by column into classification feature vectors; performing vector norm Hilbert probability space transformation on the classification feature vectors according to the following formula to obtain optimized classification feature vectors; where the formula is:

[0114]

[0115] Where V is the classification feature vector, ‖V‖2 represents the second norm of the classification feature vector, represents the square of the second norm of the classification feature vector, v i is the i-th eigenvalue of the classification feature vector, exp(·) represents the exponential operation of the vector, and the exponential operation of the vector represents calculating the natural exponential function values with the eigenvalues at each position in the vector as exponents, and v i ′ is the i-th eigenvalue of the optimized classification feature vector; and, reconstructing the optimized classification feature vectors into an optimized classification feature matrix.

[0116] In one example, in the above AR-based vehicle speed assistance method, step S170 includes: expanding the classification feature matrix into classification feature vectors based on row vectors or column vectors; performing fully connected encoding on the classification feature vectors using multiple fully connected layers of the classifier to obtain encoded classification feature vectors; and, passing the encoded classification feature vectors through the Softmax classification function of the classifier to obtain the classification result.

[0117] In summary, the AR-based vehicle speed assistance method according to the embodiments of the present application is elucidated. By using a neural network model based on deep learning, it mines the complex mapping relationship between the relative temporal changes in the vehicle distance in the detected image of the vehicle in front and the changes in the vehicle speed value, so as to assist in warning whether a vehicle collision will occur for the current vehicle speed, thereby ensuring the driving safety of the driver.

[0118] Exemplary electronic device

[0119] Next, with reference to Figure 9 the electronic device according to the embodiments of the present application will be described.

[0120] Figure 9 The block diagram of the electronic device according to the embodiments of the present application is illustrated.

[0121] As Figure 9 shown, the electronic device 10 includes one or more processors 11 and a memory 12.

[0122] The processor 11 can be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 10 to perform desired functions.

[0123] The memory 12 can include one or more computer program products, and the computer program products can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory can include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory can include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 11 can run the program instructions to implement the functions in the AR-based vehicle speed assistance system of the various embodiments of the present application described above and / or other desired functions. Various contents such as an optimized classification feature matrix can also be stored in the computer-readable storage media.

[0124] In one example, the electronic device 10 can further include: an input device 13 and an output device 14, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0125] The input device 13 can include, for example, a keyboard, a mouse, etc.

[0126] The output device 14 can output various information to the outside, including classification results, etc. The output device 14 can include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0127] Of course, for simplicity, Figure 9 only some of the components related to this application in the electronic device 10 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device 10 may further include any other appropriate components.

[0128] Exemplary computer program product and computer-readable storage medium

[0129] In addition to the above methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions that, when run by a processor, cause the processor to execute the steps in the functions of the AR-based vehicle speed assistance method according to various embodiments of the present application described in the above "Exemplary System" section of this specification.

[0130] The computer program product can be written in any combination of one or more programming languages for programming code to perform the operations of the embodiments of the present application. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, executed as an independent software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0131] Furthermore, an embodiment of the present application may also be a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are run by a processor, the processor is caused to execute the steps in the functions of the AR-based vehicle speed assistance method according to various embodiments of the present application described in the above "Exemplary System" section of this specification.

[0132] The computer-readable storage medium may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0133] The basic principles of the present application have been described in conjunction with specific embodiments. However, it should be noted that the advantages, benefits, effects, etc. mentioned in the present application are only examples and not limitations. It cannot be considered that these advantages, benefits, effects, etc. are essential for each embodiment of the present application. Additionally, the specific details disclosed above are only for illustrative and easy-to-understand purposes, not limitations. The above details do not limit the present application to necessarily adopt the above specific details for implementation.

[0134] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present application are only illustrative examples and do not intend to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open-ended terms, meaning "including but not limited to", and can be used interchangeably with each other. The word "or" and "and" used herein refer to the word "and / or", and can be used interchangeably with each other, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to", and can be used interchangeably with each other.

[0135] It should also be noted that in the devices, equipment, and methods of the present application, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present application.

[0136] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects are very obvious to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

[0137] The above description has been given for purposes of illustration and description. In addition, this description does not intend to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions, and sub-combinations thereof.

Claims

1. An AR-based vehicle speed assistance system, characterized in that, Including: A data acquisition module, configured to obtain front vehicle detection images at multiple predetermined time points within a predetermined time period collected by a camera and vehicle speed values at the multiple predetermined time points; An image feature extraction module, configured to respectively pass the front vehicle detection images at the multiple predetermined time points through a convolutional neural network model as a filter to obtain multiple front vehicle detection feature vectors; A temporal correlation encoding module, configured to pass the multiple front vehicle detection feature vectors through a context encoder based on a transformer to obtain a front vehicle relative position temporal feature vector; A vehicle speed temporal feature extraction module, configured to arrange the vehicle speed values at the multiple predetermined time points into a vehicle speed input vector according to the time dimension and then pass it through a multi-scale neighborhood feature extraction module to obtain a vehicle speed temporal feature vector; An association module, configured to perform association encoding on the vehicle speed temporal feature vector and the front vehicle relative position temporal feature vector to obtain a classification feature matrix; An optimization module, configured to perform feature distribution modulation on the classification feature matrix to obtain an optimized classification feature matrix; An early warning module, configured to pass the optimized classification feature matrix through a classifier to obtain a classification result, and the classification result is used to indicate whether a vehicle speed early warning prompt is generated; And A display module, configured to, in response to the classification result indicating that a vehicle speed early warning prompt is generated, display the vehicle speed early warning prompt on an in-vehicle screen; The vehicle speed temporal feature extraction module includes: A first neighborhood scale feature extraction unit, configured to input the vehicle speed input vector into a first convolutional layer of the multi-scale neighborhood feature extraction module to obtain a first neighborhood scale vehicle speed temporal feature vector, where the first convolutional layer has a first one-dimensional convolutional kernel with a first length; A second neighborhood scale feature extraction unit, configured to input the vehicle speed input vector into a second convolutional layer of the multi-scale neighborhood feature extraction module to obtain a second neighborhood scale vehicle speed temporal feature vector, where the second convolutional layer has a second one-dimensional convolutional kernel with a second length, and the first length is different from the second length; and A multi-scale fusion unit, configured to concatenate the first neighborhood scale vehicle speed temporal feature vector and the second neighborhood scale vehicle speed temporal feature vector to obtain the vehicle speed temporal feature vector; Wherein, the first neighborhood scale feature extraction unit is configured to: use the first convolutional layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolutional encoding on the vehicle speed input vector according to the following formula to obtain a first neighborhood scale vehicle speed temporal feature vector; Wherein, the formula is: Among them, a is the width of the first convolution kernel in the x direction, is the first convolution kernel parameter vector, is the local vector matrix for the convolution kernel function operation, w is the size of the first convolution kernel, X represents the vehicle speed input vector; and The second neighborhood scale feature extraction unit is configured to: use the second convolutional layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolutional encoding on the vehicle speed input vector according to the following formula to obtain the second neighborhood scale vehicle speed temporal feature vector; Wherein, the formula is: where b is the width of the second convolution kernel in x the direction, is the second convolution kernel parameter vector, is the local vector matrix for the convolution kernel function operation, m is the size of the second convolution kernel, X represents the vehicle speed input vector.

2. The AR-based vehicle speed assistance system according to claim 1, wherein The image feature extraction module is configured to: use each layer of the convolutional neural network model as a filter to respectively perform the following operations on the input data during the forward pass of the layer: Perform convolutional processing on the input data to obtain a convolutional feature map; Perform pooling based on a feature matrix on the convolutional feature map to obtain a pooled feature map; And Perform non - linear activation on the pooled feature map to obtain an activated feature map; Among them, the output of the last layer of the convolutional neural network serving as a filter is the multiple front - vehicle detection feature vectors, and the input of the first layer of the convolutional neural network serving as a filter is the front - vehicle detection images at the multiple predetermined time points.

3. The AR-based vehicle speed assistance system according to claim 2, wherein, The temporal correlation encoding module includes: A context encoding unit for performing global context semantic encoding based on the Transformer idea on the multiple front - vehicle detection feature vectors using a Transformer of a context encoder including an embedding layer to obtain multiple global context semantic front - vehicle detection feature vectors; and A concatenation unit for concatenating the multiple global context semantic front - vehicle detection feature vectors to obtain the front - vehicle relative position temporal feature vector.

4. The AR-based vehicle speed assistance system according to claim 3, wherein The context encoding unit includes: A query vector construction sub - unit for arranging the multiple front - vehicle detection feature vectors in one - dimension to obtain a global front - vehicle detection feature vector; A self - attention sub - unit for calculating the product between the global front - vehicle detection feature vector and the transposed vectors of each of the multiple front - vehicle detection feature vectors to obtain multiple self - attention correlation matrices; A normalization sub - unit for respectively performing normalization processing on each of the multiple self - attention correlation matrices to obtain multiple normalized self - attention correlation matrices; An attention degree calculation sub - unit for passing each of the multiple normalized self - attention correlation matrices through a Softmax classification function to obtain multiple probability values; An attention application sub - unit for respectively weighting each of the multiple front - vehicle detection feature vectors with each of the probability values in the multiple probability values to obtain multiple context semantic front - vehicle detection feature vectors; A concatenation sub - unit for concatenating the multiple context semantic front - vehicle detection feature vectors to obtain the global context semantic front - vehicle detection feature vector.

5. The AR-based vehicle speed assistance system according to claim 4, wherein, The multi - scale neighborhood feature extraction module includes: a first convolutional layer, a second convolutional layer parallel to the first convolutional layer, and a multi - scale feature fusion layer connected to the first convolutional layer and the second convolutional layer, where the first convolutional layer uses a one - dimensional convolutional kernel with a first length, and the second convolutional layer uses a one - dimensional convolutional kernel with a second length.

6. The AR-based vehicle speed assistance system according to claim 1, characterized in that The association module is used to: perform association encoding on the vehicle speed temporal feature vector and the front - vehicle relative position temporal feature vector according to the following formula to obtain a classification feature matrix; where the formula is: wherein represents the vehicle speed time series feature vector, represents the transposed vector of the vehicle speed time series feature vector, represents the relative position time series feature vector of the vehicle ahead, represents the classification feature matrix, represents vector multiplication.

7. The AR-based vehicle speed assistance system according to claim 6, wherein The optimization module includes: An unfolding unit for unfolding the classification feature matrix row - by - row or column - by - column into a classification feature vector; A feature optimization unit for performing vector - norm Hilbert probability space transformation on the classification feature vector according to the following formula to obtain an optimized classification feature vector; where the formula is: wherein is the classification feature vector, represents the two-norm of the classification feature vector, represents the square of the two-norm of the classification feature vector, is the -th eigenvalue of the classification feature vector, represents the exponential operation of a vector, and the exponential operation of the vector means calculating the natural exponential function values with the eigenvalues at each position in the vector as exponents, and is the -th eigenvalue of the optimized classification feature vector; and A matrix reconstruction unit for reconstructing the optimized classification feature vector into the optimized classification feature matrix.

8. The AR-based vehicle speed assistance system according to claim 7, wherein The warning module includes: A classification feature vector generation unit, configured to expand the classification feature matrix into a classification feature vector based on row vectors or column vectors; A fully-connected encoding unit, configured to perform fully-connected encoding on the classification feature vector using multiple fully-connected layers of the classifier to obtain an encoded classification feature vector; and A classification result generation unit, configured to pass the encoded classification feature vector through the Softmax classification function of the classifier to obtain the classification result.

9. An AR-based vehicle speed assistance method, characterized in that, Comprising: Obtaining front vehicle detection images at multiple predetermined time points within a predetermined time period collected by a camera and vehicle speed values at the multiple predetermined time points; Respectively passing the front vehicle detection images at the multiple predetermined time points through a convolutional neural network model serving as a filter to obtain multiple front vehicle detection feature vectors; Passing the multiple front vehicle detection feature vectors through a context encoder based on a transformer to obtain a front vehicle relative position time series feature vector; Arranging the vehicle speed values at the multiple predetermined time points in a time dimension to form a vehicle speed input vector, and then passing it through a multi-scale neighborhood feature extraction module to obtain a vehicle speed time series feature vector; Performing correlation encoding on the vehicle speed time series feature vector and the front vehicle relative position time series feature vector to obtain a classification feature matrix; Performing feature distribution modulation on the classification feature matrix to obtain an optimized classification feature matrix; Passing the classification feature matrix through a classifier to obtain a classification result, where the classification result is used to indicate whether a vehicle speed warning prompt is generated; And In response to the classification result indicating that a vehicle speed warning prompt is generated, displaying the vehicle speed warning prompt on an in-vehicle screen; The vehicle speed time series feature extraction module includes: A first neighborhood scale feature extraction unit, configured to input the vehicle speed input vector into a first convolutional layer of the multi-scale neighborhood feature extraction module to obtain a first neighborhood scale vehicle speed time series feature vector, where the first convolutional layer has a first one-dimensional convolutional kernel with a first length; A second neighborhood scale feature extraction unit, configured to input the vehicle speed input vector into a second convolutional layer of the multi-scale neighborhood feature extraction module to obtain a second neighborhood scale vehicle speed time series feature vector, where the second convolutional layer has a second one-dimensional convolutional kernel with a second length, and the first length is different from the second length; and A multi-scale fusion unit, configured to concatenate the first neighborhood scale vehicle speed time series feature vector and the second neighborhood scale vehicle speed time series feature vector to obtain the vehicle speed time series feature vector; Wherein, the first neighborhood scale feature extraction unit is configured to: use the first convolutional layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolutional encoding on the vehicle speed input vector according to the following formula to obtain a first neighborhood scale vehicle speed time series feature vector; Wherein, the formula is: Among them, a is the width of the first convolution kernel in the x direction, is the first convolution kernel parameter vector, is the local vector matrix for the convolution kernel function operation, w is the size of the first convolution kernel, X represents the vehicle speed input vector; and The second neighborhood scale feature extraction unit is configured to: use the second convolutional layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolutional encoding on the vehicle speed input vector according to the following formula to obtain the second neighborhood scale vehicle speed time series feature vector; Wherein, the formula is: where b is the width of the second convolution kernel in x the direction, is the second convolution kernel parameter vector, X is the local vector matrix for the convolution kernel function operation, m is the size of the second convolution kernel, represents the vehicle speed input vector.

Images