Clock signal prediction method and device and vehicle

Abstract

The invention provides a clock signal prediction method and device and a vehicle. The method comprises the steps that a surrounding environment image of the vehicle and three-dimensional geometric constraint information between the vehicle and the surrounding environment are acquired; extracting a plurality of image features from the surrounding environment image; fusing each image feature and the three-dimensional geometric constraint information to obtain a fused feature; based on the fused feature, a clock signal of the vehicle is predicted. According to the method, after the image features and the three-dimensional geometric constraint information are fused, the obtained fusion features comprise visual information and spatial information, and then the clock signal can be accurately predicted based on the fusion features. In addition, the surrounding environment image and the three-dimensional geometric constraint information can be obtained at low cost, and thus the equipment cost is saved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent vehicle technology, and in particular to a clock signal prediction method, device and vehicle. Background Art

[0002] In intelligent vehicle systems, clock signal prediction usually refers to predicting the clock state or the time when an event will occur in the future, such as predicting the time when a vehicle arrives at a certain location, the time when the vehicle collides with an obstacle in front, etc.

[0003] At present, laser radar is mostly used to obtain high-precision three-dimensional information and generate point cloud data, and stereo cameras are used to obtain depth information and generate depth images, and clock signal prediction is performed by combining point cloud data and depth images. However, laser radar and stereo cameras are expensive and difficult to be applied to ordinary vehicles on a large scale. In addition, the use of laser radar and stereo cameras for clock signal prediction also requires the development of complex data fusion algorithms, which increases the difficulty of design and development. Summary of the invention

[0004] The present invention provides a clock signal prediction method, device and vehicle to solve the defects in the prior art.

[0005] The present invention provides a clock signal prediction method, comprising the following steps: Acquire a surrounding environment image of the vehicle and three-dimensional geometric constraint information between the vehicle and the surrounding environment; Extracting multiple image features from the surrounding environment image; Fusing each image feature and the three-dimensional geometric constraint information to obtain a fused feature; Based on the fused features, a clock signal of the vehicle is predicted.

[0006] According to a clock signal prediction method provided by the present invention, extracting multiple image features from the surrounding environment image, fusing the image features and the three-dimensional geometric constraint information to obtain fused features, and predicting the clock signal of the vehicle based on the fused features, including: Based on the clock signal prediction model, extract multiple image features from the surrounding environment image, fuse the image features and the three-dimensional geometric constraint information to obtain fused features, and predict the clock signal of the vehicle based on the fused features; The clock signal prediction model is trained based on the sample surrounding environment image, the sample three-dimensional geometric constraint information and the corresponding clock signal label.

[0007] According to a clock signal prediction method provided by the present invention, the training step of the clock signal prediction model includes: Extracting a plurality of sample image features from the sample surrounding environment image; Fusing the image features of each sample and the three-dimensional geometric constraint information of the sample to obtain a sample fusion feature; Based on the features of each sample image, predict the category of the sample's surrounding environment elements; Based on the sample fusion feature, predicting a sample clock signal; Based on the difference between the sample surrounding environment element category and the surrounding environment element label, and the difference between the sample clock signal and the clock signal label, the initial model is trained to obtain the clock signal prediction model.

[0008] According to a clock signal prediction method provided by the present invention, the initial model is trained based on the difference between the sample surrounding environment element category and the surrounding environment element label, and the difference between the sample clock signal and the clock signal label to obtain the clock signal prediction model, including: Determining a classification loss based on a difference between the sample surrounding environment element category and the surrounding environment element label; determining a prediction loss based on a difference between the sample clock signal and the clock signal label; Based on preset weights, the prediction loss and the classification loss are weightedly summed to obtain a mixed loss, and the initial model is trained based on the mixed loss to obtain the clock signal prediction model.

[0009] According to a clock signal prediction method provided by the present invention, the fusing of each image feature and the three-dimensional geometric constraint information to obtain a fused feature includes: Determining a weight of each image feature based on a similarity between each image feature and the three-dimensional geometric constraint information; Based on the weight of each image feature, each image feature is weighted and summed to obtain the fusion feature.

[0010] According to a clock signal prediction method provided by the present invention, the predicting of the clock signal of the vehicle further comprises: After removing the abnormal signal in the clock signal, the noise signal in the clock signal is filtered out.

[0011] The present invention also provides a clock signal prediction device, comprising the following modules: An acquisition unit, used to acquire an image of the surrounding environment of the vehicle and three-dimensional geometric constraint information between the vehicle and the surrounding environment; An extraction unit, used for extracting a plurality of image features from the surrounding environment image; A fusion unit, used for fusing each image feature and the three-dimensional geometric constraint information to obtain a fusion feature; A detection unit is used to predict a clock signal of the vehicle based on the fused features.

[0012] The present invention also provides a vehicle, comprising: the clock signal prediction device as described above.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the clock signal prediction methods described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the clock signal prediction method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the clock signal prediction method described above is implemented.

[0016] The clock signal prediction method, device and vehicle provided by the present invention have image features that can effectively represent the visual information of the surrounding environment, and the three-dimensional geometric constraint information provides the distance information and spatial position information missing in the image features, so that after fusing the image features and the three-dimensional geometric constraint information, the fused features obtained contain visual information and spatial information, and the clock signal can be accurately predicted based on the fused features. In addition, the present invention can obtain the surrounding environment image and the three-dimensional geometric constraint information at a low cost, thereby saving equipment costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the clock signal prediction method provided by the present invention.

[0019] Figure 2 It is a flow chart of the clock signal prediction model training method provided by the present invention.

[0020] Figure 3 It is a flowchart diagram of an implementation of step 130 in the clock signal prediction method provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the clock signal prediction device provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] At present, related technologies mostly use laser radar to obtain high-precision three-dimensional information and generate point cloud data, and use stereo cameras to obtain depth information and generate depth images, and combine point cloud data and depth images to predict clock signals. However, laser radar and stereo cameras are expensive and difficult to be applied to ordinary vehicles on a large scale. In addition, the use of laser radar and stereo cameras for clock signal prediction also requires the development of complex data fusion algorithms, which increases the difficulty of design and development.

[0025] In addition, related technologies also use monocular cameras for clock signal prediction, that is, extracting image features from the two-dimensional image captured by the monocular camera, and predicting the clock signal based on the image features. However, in complex road scenes, the accuracy and reliability of image feature extraction are limited, which in turn affects the prediction accuracy of the clock signal.

[0026] Figure 1 FIG. 1 is a flow chart of the clock signal prediction method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0027] Step 110: Acquire the surrounding environment image of the vehicle and the three-dimensional geometric constraint information between the vehicle and the surrounding environment.

[0028] Here, the surrounding environment image refers to a two-dimensional image containing real-time visual information around the vehicle, which is used to obtain the state of the surrounding environment, such as other vehicles, pedestrians, traffic signs, road conditions, etc. The surrounding environment image can be acquired by a low-cost camera, which can be an ordinary RGB camera or a monocular camera used to capture images.

[0029] The three-dimensional geometric constraint information is used to characterize the spatial relationship between the vehicle and the surrounding environment. The three-dimensional geometric constraint information may include the relative position between the vehicle and the surrounding environment, the three-dimensional structure information of the surrounding environment, etc. The three-dimensional geometric constraint information may be determined by information obtained through the Global Positioning System (GPS), Inertial Measurement Unit (IMU), high-precision map, etc.

[0030] For example, the vehicle's position and speed can be obtained through GPS, and the vehicle's acceleration, angular velocity, and posture can be obtained through IMU. The vehicle's three-dimensional trajectory information can be constructed based on the vehicle's position, speed, acceleration, angular velocity, and posture. The three-dimensional combination constraint information between the vehicle and the surrounding environment can be determined based on the vehicle's three-dimensional trajectory information and the three-dimensional structure information of the surrounding environment in the high-precision map.

[0031] It can be seen that the embodiments of the present invention can obtain the vehicle's surrounding environment image through a low-cost camera, and obtain the three-dimensional geometric constraint information between the vehicle and the surrounding environment through low-cost GPS, IMU and high-precision map, that is, use low-cost equipment to obtain the above information, saving equipment costs.

[0032] Step 120: extract multiple image features from the surrounding environment image.

[0033] Specifically, the image features are used to characterize key information in the surrounding environment image, and the image features may be features related to the road environment, such as road signs, lane lines, obstacles, etc. Among them, an image feature extraction model may be used to extract multiple image features from the surrounding environment image, and the image feature extraction model may be constructed based on CNN.

[0034] Step 130: fuse the image features and the three-dimensional geometric constraint information to obtain a fused feature.

[0035] Step 140: predict the clock signal of the vehicle based on the fused features.

[0036] Specifically, image features can effectively represent the visual information of the surrounding environment (such as color, shape, texture, etc.), so as to accurately identify different objects in the surrounding environment. However, image features provide two-dimensional information, which is limited in distance perception and spatial positioning, while three-dimensional geometric constraint information provides the distance information and spatial position information missing in image features. On this basis, the image features and three-dimensional geometric constraint information are fused, so that the obtained fused features not only contain the visual information of the surrounding environment, but also can accurately reflect the spatial relationship between the vehicle and the surrounding environment.

[0037] On this basis, based on the visual information and spatial information contained in the fused features, the clock signal of the vehicle can be accurately predicted. Here, the clock signal can be understood as a signal related to the dynamic state of the vehicle within a specific time. For example, the clock signal can be the exact moment when the vehicle arrives at a certain location, or the moment when the vehicle collides with the obstacle in front.

[0038] After obtaining the clock signal, the clock signal can be applied to various environments of autonomous driving, such as target detection, path planning, motion prediction, etc. based on the clock signal. For example, if the clock signal is the collision moment between the vehicle and the obstacle ahead, the path planning can be re-performed based on the clock signal to enable the vehicle to avoid the obstacle ahead.

[0039] In the clock signal prediction method provided by the embodiment of the present invention, the image features can effectively represent the visual information of the surrounding environment, and the three-dimensional geometric constraint information provides the distance information and spatial position information missing in the image features, so that after fusing the image features and the three-dimensional geometric constraint information, the fused features obtained contain the visual information and the spatial information, and thus the clock signal can be accurately predicted based on the fused features. In addition, the embodiment of the present invention can obtain the surrounding environment image and the three-dimensional geometric constraint information at a low cost, thereby saving the equipment cost.

[0040] Based on the above embodiment, multiple image features are extracted from the surrounding environment image, and each image feature and three-dimensional geometric constraint information are fused to obtain a fused feature. Based on the fused feature, the clock signal of the vehicle is predicted, including: Based on the clock signal prediction model, multiple image features are extracted from the surrounding environment image, and the image features and three-dimensional geometric constraint information are fused to obtain fused features. Based on the fused features, the clock signal of the vehicle is predicted; The clock signal prediction model is trained based on the sample's surrounding environment image, the sample's three-dimensional geometric constraint information, and the corresponding clock signal label.

[0041] As an optional embodiment, the clock prediction model may include a feature extraction module, an information fusion module and a signal prediction module. The feature extraction module is used to extract multiple image features from the surrounding environment image, the information fusion module is used to fuse each image feature and three-dimensional geometric constraint information to obtain a fusion feature, and the signal prediction module is used to predict the clock signal of the vehicle based on the fusion feature.

[0042] Among them, the feature extraction module can use CNN to extract multiple image features. Among them, CNN can effectively capture local and global features in the image through the stacking of multiple convolutional layers and pooling layers, and the extracted image features include both detail information and global information. Among them, the convolution layer is used to extract image features at different levels. For example, a convolution layer can be constructed using a 3×3 or 5×5 convolution kernel. The pooling layer is used to reduce the dimension of the image features extracted by the convolution layer to retain the main features. Commonly used pooling methods include maximum pooling and average pooling. In addition, CNN uses the ReLU activation function to increase the nonlinear expression ability of the network. Optionally, the convolution layer design of CNN can focus on extracting features related to the road environment, such as road signs, lane lines, obstacles, etc., so that vehicle positioning and path planning can be assisted based on these features.

[0043] In addition, the feature extraction module can be obtained by fine-tuning the pre-trained model to accelerate model convergence and improve the accuracy of feature extraction. The pre-trained model can be a CNN trained on a large-scale image dataset (such as ImageNet). By training on a large-scale image dataset, CNN can learn common low-level and high-level image features, such as edges, textures, shapes, etc. In this case, the pre-trained model has mastered rich visual features, so when migrating to other tasks, it can quickly adapt to new scenarios and improve the effect of feature extraction.

[0044] The information fusion module can use a multilayer perceptron (MLP) or an attention mechanism to fuse the features of each image and the three-dimensional geometric constraint information to obtain fused features. Each layer of MLP can be followed by a ReLU activation function to increase the expressive power of the model. The attention mechanism dynamically adjusts the weights by calculating the similarity between the image features and the three-dimensional geometric constraint information, making the fusion result more accurate.

[0045] Before predicting the clock signal of the vehicle based on the clock signal prediction model, the clock signal prediction model can also be pre-trained, which can be achieved by executing the following steps: First, a large number of sample surrounding environment images and sample three-dimensional geometric constraint information are collected, and their corresponding clock signal labels are determined through manual annotation. Then, the initial model is trained based on the sample surrounding environment images, sample three-dimensional geometric constraint information and corresponding clock signal labels to obtain the clock signal prediction model.

[0046] As an optional embodiment, after the clock signal prediction model is trained, the sample surrounding environment image, the sample three-dimensional geometric constraint information and the corresponding clock signal label can be incrementally obtained as a test set, and the performance of the clock signal prediction model can be verified based on the test set. If the performance does not meet the requirements, the sample surrounding environment image, the sample three-dimensional geometric constraint information and the corresponding clock signal label are continuously obtained to update the parameters of the clock signal prediction model.

[0047] Based on any of the above embodiments, Figure 2 FIG. 1 is a flow chart of the clock signal prediction model training method provided by the present invention, such as Figure 2 As shown, the training steps of the clock signal prediction model include: Step 210: extract multiple sample image features from the sample surrounding environment image.

[0048] Here, the sample surrounding environment image refers to a two-dimensional image containing real-time visual information around the vehicle. This visual information is used to obtain the state of the surrounding environment, such as the state of other vehicles, pedestrians, traffic signs, road conditions, etc.

[0049] The sample image features are used to characterize key information in the sample surrounding environment image, and the sample image features may be features related to the road environment, such as road signs, lane lines, obstacles, etc. Among them, an image feature extraction model may be used to extract the sample image features from the sample surrounding environment image, and the image feature extraction model may be constructed based on CNN.

[0050] As an optional embodiment, the sample surrounding environment image can be obtained by data enhancement of the original collected surrounding environment image, wherein the data enhancement methods may include image rotation, image cropping, image flipping, color transformation, etc. Image rotation is to randomly rotate the image by a certain angle (such as -15° to 15°) to increase the road features at different angles and improve the model's adaptability to rotation invariance. Image cropping is to randomly crop a part of the image and then adjust it to the original size to simulate different camera perspectives and enhance the robustness of the model. Image flipping is to flip the image horizontally or vertically to improve the model's recognition ability for mirrored environments. Color transformation is to adjust the brightness, contrast, saturation, etc. of the image to simulate different lighting conditions and improve the performance of the model in various environments.

[0051] Step 220: fuse the image features of each sample and the three-dimensional geometric constraint information of the sample to obtain the sample fusion feature.

[0052] Specifically, the sample image features can effectively represent the visual information of the surrounding environment (such as color, shape, texture, etc.), so as to accurately identify different objects in the surrounding environment. However, the sample image features provide two-dimensional information, which is limited in distance perception and spatial positioning, while the sample three-dimensional geometric constraint information provides the distance information and spatial position information missing in the image features. On this basis, the sample image features and the sample three-dimensional geometric constraint information are fused, so that the obtained sample fusion features can not only contain the visual information of the surrounding environment, but also accurately reflect the spatial relationship between the vehicle and the surrounding environment.

[0053] Step 230: predict the category of the sample's surrounding environment elements based on the features of each sample image.

[0054] Specifically, the category of the sample surrounding environment element can be understood as the category to which each element in the sample surrounding environment image belongs, wherein the category to which each element in the sample surrounding environment image belongs may include road signs, lane lines, obstacles, etc.

[0055] The image features of each sample are used to characterize the visual information of each element in the surrounding environment of the sample, so that based on the visual information of each element, different elements can be distinguished, that is, the categories to which different elements belong can be determined.

[0056] Step 240: predict the sample clock signal based on the sample fusion feature.

[0057] Specifically, the sample fusion feature not only contains the visual information of the surrounding environment, but also can accurately reflect the spatial relationship between the vehicle and the surrounding environment. On this basis, based on the visual information and spatial information contained in the sample fusion feature, the sample clock signal can be accurately predicted.

[0058] Step 250: Based on the difference between the sample surrounding environment element category and the surrounding environment element label, and the difference between the sample clock signal and the clock signal label, the initial model is trained to obtain a clock signal prediction model.

[0059] Specifically, the difference between the sample surrounding environment element category and the surrounding environment element label is used to train the surrounding environment element recognition task, and the difference between the sample clock signal and the clock signal label is used to train the clock signal prediction task. The embodiment of the present invention trains the initial model based on the difference between the sample surrounding environment element category and the surrounding environment element label, and the difference between the sample clock signal and the clock signal label, that is, the surrounding environment element recognition task and the clock signal prediction task are collaboratively trained, so as to better capture the key information in the surrounding environment and enhance the expression ability of the fusion feature.

[0060] In addition, the collaborative training of the two tasks can improve the generalization ability of the model and effectively reduce the risk of overfitting. If one task is trained alone, the model may overfit the data of a specific task, while the collaborative training of the two tasks can optimize multiple objectives at the same time, making the model more robust and more stable in actual scenarios.

[0061] The task of predicting clock signals and the task of identifying surrounding environmental elements are related. For example, accurately identifying lane lines and obstacles helps predict accurate clock signals, because the behavior of the vehicle is affected by the surrounding road environment. Therefore, the collaborative training of the two tasks can more comprehensively understand the entire driving scenario, which is more holistic and practical than single-task training.

[0062] It can be seen that the collaborative training of the two tasks in the embodiment of the present invention can not only complete the two tasks at the same time, but also complement and share features to improve the prediction accuracy and robustness of the overall model. Compared with predicting the clock signal and identifying the surrounding environment elements separately, it can better understand complex driving scenarios and make more accurate decisions.

[0063] In addition, the embodiment of the present invention can also use the Adam optimizer to dynamically adjust the learning rate of each parameter in the model, and the learning rate scheduling mechanism can include cosine annealing, step descent, etc. The Adam optimizer can improve the convergence speed of the model, and the learning rate scheduling mechanism can avoid overfitting and underfitting of the model, thereby improving the training effect and stability of the model.

[0064] Based on any of the above embodiments, based on the difference between the sample surrounding environment element category and the surrounding environment element label, and the difference between the sample clock signal and the clock signal label, the initial model is trained to obtain a clock signal prediction model, including: Determine the classification loss based on the difference between the category of the surrounding environment elements of the sample and the label of the surrounding environment elements; determining a prediction loss based on a difference between the sample clock signal and the clock signal label; Based on the preset weights, the prediction loss and the classification loss are weighted and summed to obtain the mixed loss, and the initial model is trained based on the mixed loss to obtain the clock signal prediction model.

[0065] Specifically, the smaller the difference between the sample surrounding environment element category and the surrounding environment element label, the smaller the corresponding classification loss, and thus the better the training effect of the surrounding environment element recognition task. Among them, the cross entropy loss can be determined based on the difference between the sample surrounding environment element category and the surrounding environment element label, and the cross entropy loss can be used as the classification loss.

[0066] The smaller the difference between the sample clock signal and the clock signal label, the smaller the corresponding prediction loss, and thus the better the training effect of the clock signal prediction task. The mean square error can be determined based on the difference between the sample clock signal and the clock signal label, and the prediction loss can be determined based on the mean square error.

[0067] After determining the prediction loss and the classification loss, based on the preset weights, the prediction loss and the classification loss are weighted and summed to obtain a mixed loss, and the initial model is trained based on the mixed loss to obtain a clock signal prediction model. The mixed loss can be determined based on the following formula:

[0068] in, represents the mixing loss, represents the prediction loss, represents the classification loss, α and β is the weight coefficient.

[0069] Based on any of the above embodiments, Figure 3 is a flow chart of an implementation of step 130 in the clock signal prediction method provided by the present invention, such as Figure 3 As shown, step 130 fuses each image feature and the three-dimensional geometric constraint information to obtain a fused feature, including: Step 131: determining a weight of each image feature based on the similarity between each image feature and the three-dimensional geometric constraint information; Step 132: Based on the weights of the image features, perform weighted summation on the image features to obtain a fusion feature.

[0070] Specifically, the similarity between each image feature and the 3D geometric constraint information is used to characterize the contribution of each image feature to the 3D geometric size information. The higher the similarity, the greater the contribution of the corresponding image feature to the 3D geometric size information, and thus the greater its weight. Image features with high weights occupy a larger proportion in the weighted sum, and thus can provide more 3D geometric size information, that is, the fusion features obtained by weighted summation not only contain the visual information carried by the image features, but also contain rich 3D geometric size information.

[0071] As an optional embodiment, the similarity between each image feature and the three-dimensional geometric constraint information can be calculated, and the attention weight of each image feature can be determined based on the similarity. Based on the attention weight of each image feature, each image feature is weighted and summed to generate a fusion feature.

[0072] The embodiments of the present invention can dynamically determine the attention weights corresponding to image features according to the similarities between different image features and three-dimensional geometric constraint information, thereby improving the flexibility and accuracy of feature fusion and enhancing the adaptability of the model to complex environments.

[0073] Based on any of the above embodiments, predicting a clock signal of a vehicle, and then further comprising: After removing the abnormal signal in the clock signal, the noise signal in the clock signal is filtered out.

[0074] Specifically, considering that there may be abnormal signals and noise signals in the clock signal, in order to ensure that the predicted clock signal has high stability and accuracy, the embodiment of the present invention removes the abnormal signals in the clock signal and then filters out the noise signals in the clock signal.

[0075] Among them, statistical methods and machine learning algorithms (such as isolation forest) can be used to identify and eliminate abnormal signals. Among them, the isolation forest algorithm can effectively detect abnormal signals by calculating the degree of isolation of data points. In order to ensure the stability and accuracy of the clock signal, after eliminating the abnormal signal in the clock signal, the Kalman filter algorithm can be used to smooth the clock signal to filter out the noise signal in the clock signal. Among them, the Kalman filter is a recursive estimation algorithm that can effectively filter the noisy clock signal through prediction and update steps to generate a smooth clock signal.

[0076] Among them, the clock signal after filtering out noise can be used for precise control of the vehicle, such as predicting when the vehicle will reach a certain location or obstacle, and adjusting the speed or braking accordingly to optimize path planning. It can also provide a time reference for the autonomous driving system to help the system perform operations at the right time, such as changing lanes, turning or accelerating. In scenarios where multiple vehicles or fleets are working together, accurate clock signals can coordinate time synchronization between vehicles to ensure efficient and safe traffic scheduling.

[0077] The clock signal prediction device provided by the present invention is described below. The clock signal prediction device described below and the clock signal prediction method described above can be referenced to each other.

[0078] Based on any of the above embodiments, Figure 4 is a schematic diagram of the structure of the clock signal prediction device provided by the present invention, such as Figure 4 As shown, the device comprises: The acquisition unit 410 is used to acquire the surrounding environment image of the vehicle and the three-dimensional geometric constraint information between the vehicle and the surrounding environment; An extraction unit 420, configured to extract a plurality of image features from the surrounding environment image; A fusion unit 430 is used to fuse the image features and the three-dimensional geometric constraint information to obtain a fusion feature; The detection unit 440 is used to predict the clock signal of the vehicle based on the fused features.

[0079] Based on any of the above embodiments, extracting multiple image features from the surrounding environment image, fusing the image features and the three-dimensional geometric constraint information to obtain a fused feature, and predicting the clock signal of the vehicle based on the fused feature, including: Based on the clock signal prediction model, multiple image features are extracted from the surrounding environment image, and the image features and three-dimensional geometric constraint information are fused to obtain fused features. Based on the fused features, the clock signal of the vehicle is predicted; The clock signal prediction model is trained based on the sample's surrounding environment image, the sample's three-dimensional geometric constraint information, and the corresponding clock signal label.

[0080] Based on any of the above embodiments, the training step of the clock signal prediction model includes: Extracting multiple sample image features from the sample surrounding environment image; Fuse the image features of each sample and the three-dimensional geometric constraint information of the sample to obtain the sample fusion feature; Based on the features of each sample image, predict the category of the sample's surrounding environment elements; Based on the sample fusion features, the sample clock signal is predicted; Based on the difference between the category of the sample surrounding environment element and the surrounding environment element label, and the difference between the sample clock signal and the clock signal label, the initial model is trained to obtain the clock signal prediction model.

[0081] Based on any of the above embodiments, based on the difference between the sample surrounding environment element category and the surrounding environment element label, and the difference between the sample clock signal and the clock signal label, the initial model is trained to obtain a clock signal prediction model, including: Determine the classification loss based on the difference between the category of the surrounding environment elements of the sample and the label of the surrounding environment elements; determining a prediction loss based on a difference between the sample clock signal and the clock signal label; Based on the preset weights, the prediction loss and the classification loss are weighted and summed to obtain the mixed loss, and the initial model is trained based on the mixed loss to obtain the clock signal prediction model.

[0082] Based on any of the above embodiments, the image features and the three-dimensional geometric constraint information are fused to obtain fused features, including: Determining the weight of each image feature based on the similarity between each image feature and the three-dimensional geometric constraint information; Based on the weight of each image feature, each image feature is weighted and summed to obtain the fusion feature.

[0083] Based on any of the above embodiments, predicting a clock signal of a vehicle, and then further comprising: After removing the abnormal signal in the clock signal, the noise signal in the clock signal is filtered out.

[0084] Based on any of the above embodiments, the present invention also provides a vehicle, comprising: a clock signal prediction device as described in any of the above embodiments, based on which the clock signal prediction device can accurately predict the clock signal, so that the controller in the vehicle can accurately perform path planning, motion prediction, etc. based on the clock signal.

[0085] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the clock signal prediction method, which includes: acquiring the surrounding environment image of the vehicle and the three-dimensional geometric constraint information between the vehicle and the surrounding environment; extracting multiple image features from the surrounding environment image; fusing each image feature and the three-dimensional geometric constraint information to obtain a fused feature; and predicting the clock signal of the vehicle based on the fused feature.

[0086] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0087] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the clock signal prediction method provided by the above-mentioned methods, which includes: acquiring a surrounding environment image of a vehicle, and three-dimensional geometric constraint information between the vehicle and the surrounding environment; extracting multiple image features from the surrounding environment image; fusing each image feature and the three-dimensional geometric constraint information to obtain a fused feature; and predicting the clock signal of the vehicle based on the fused feature.

[0088] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the clock signal prediction method provided by the above-mentioned methods, the method comprising: acquiring a surrounding environment image of a vehicle, and three-dimensional geometric constraint information between the vehicle and the surrounding environment; extracting multiple image features from the surrounding environment image; fusing each image feature and the three-dimensional geometric constraint information to obtain a fused feature; and predicting the clock signal of the vehicle based on the fused feature.

[0089] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A clock signal prediction method, characterized in that: include: Acquire a surrounding environment image of the vehicle and three-dimensional geometric constraint information between the vehicle and the surrounding environment; Extracting multiple image features from the surrounding environment image; Fusing each image feature and the three-dimensional geometric constraint information to obtain a fused feature; Based on the fused features, a clock signal of the vehicle is predicted.

2. The clock signal prediction method according to claim 1, characterized in that: The extracting a plurality of image features from the surrounding environment image, fusing the image features and the three-dimensional geometric constraint information to obtain a fused feature, and predicting the clock signal of the vehicle based on the fused feature, comprises: Based on the clock signal prediction model, extract multiple image features from the surrounding environment image, fuse the image features and the three-dimensional geometric constraint information to obtain fused features, and predict the clock signal of the vehicle based on the fused features; The clock signal prediction model is trained based on the sample surrounding environment image, the sample three-dimensional geometric constraint information and the corresponding clock signal label.

3. The clock signal prediction method according to claim 2, characterized in that: The training steps of the clock signal prediction model include: Extracting a plurality of sample image features from the sample surrounding environment image; Fusing the image features of each sample and the three-dimensional geometric constraint information of the sample to obtain a sample fusion feature; Based on the features of each sample image, predict the category of the sample's surrounding environment elements; Based on the sample fusion feature, predicting a sample clock signal; Based on the difference between the sample surrounding environment element category and the surrounding environment element label, and the difference between the sample clock signal and the clock signal label, the initial model is trained to obtain the clock signal prediction model.

4. The clock signal prediction method according to claim 3, characterized in that: The initial model is trained based on the difference between the sample surrounding environment element category and the surrounding environment element label, and the difference between the sample clock signal and the clock signal label to obtain the clock signal prediction model, including: Determining a classification loss based on a difference between the sample surrounding environment element category and the surrounding environment element label; determining a prediction loss based on a difference between the sample clock signal and the clock signal label; Based on preset weights, the prediction loss and the classification loss are weightedly summed to obtain a mixed loss, and the initial model is trained based on the mixed loss to obtain the clock signal prediction model.

5. The clock signal prediction method according to any one of claims 1 to 4, characterized in that: The fusing of the image features and the three-dimensional geometric constraint information to obtain the fused features includes: Determining a weight of each image feature based on a similarity between each image feature and the three-dimensional geometric constraint information; Based on the weight of each image feature, each image feature is weighted and summed to obtain the fusion feature.

6. The clock signal prediction method according to any one of claims 1 to 4, characterized in that: The predicting of the clock signal of the vehicle further comprises: After removing the abnormal signal in the clock signal, the noise signal in the clock signal is filtered out.

7. A clock signal prediction device, characterized in that: include: An acquisition unit, used to acquire an image of the surrounding environment of the vehicle and three-dimensional geometric constraint information between the vehicle and the surrounding environment; An extraction unit, used for extracting a plurality of image features from the surrounding environment image; A fusion unit, used for fusing each image feature and the three-dimensional geometric constraint information to obtain a fusion feature; A detection unit is used to predict a clock signal of the vehicle based on the fused features.

8. A vehicle, characterized in that: include: The clock signal prediction device as claimed in claim 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the clock signal prediction method according to any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the clock signal prediction method according to any one of claims 1 to 6 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the clock signal prediction method according to any one of claims 1 to 6 is implemented.

Images