Depth estimation method, device and equipment for traffic sign and vehicle

Abstract

The invention provides a traffic sign depth estimation method, device and equipment and a vehicle, and the method comprises the steps: obtaining a plurality of frames of continuous environment images, and determining the position, type and first depth information of a target object in each frame of environment image; based on the position and the category of the target object, selecting a preset number of environment images which contain the same target object and are continuous in time from multiple frames of continuous environment images; determining second depth information of the same target object in each frame of selected environment image according to a preset parameter of a time node corresponding to each frame of environment image in the preset number of environment images; and determining target depth information of the same target object according to the first depth information and the second depth information. According to the invention, the depth estimation precision of the traffic sign is improved, and the positioning accuracy of the traffic sign is improved.

Description

Technical Field

[0001] The present application relates to the technical field of depth estimation, and in particular to a method, device, equipment and vehicle for depth estimation of traffic signs. Background Art

[0002] With the rapid development of autonomous driving technology and intelligent transportation systems (ITS), the recognition and positioning of traffic signs have become an important research area in autonomous driving and traffic management. Accurate recognition and positioning of traffic signs are crucial to the safety and reliability of autonomous driving systems, especially how to accurately detect and identify traffic signs from complex traffic environment images and achieve depth estimation of traffic signs.

[0003] Existing traffic sign detection methods suffer from low target depth estimation accuracy when facing complex traffic environments. Summary of the invention

[0004] In view of the above, it is necessary to propose a method, device, equipment and vehicle for depth estimation of traffic signs to solve the technical problem that the existing traffic sign detection method has low target depth estimation accuracy when facing complex traffic environments.

[0005] In a first aspect, the present application provides a method for estimating the depth of a traffic sign, the method comprising: acquiring a plurality of continuous frames of environmental images, and determining a position, a category, and a first depth information of a target object in each frame of the environmental image; based on the position and the category of the target object, selecting a preset number of environmental images that contain the same target object and are temporally continuous from the plurality of continuous frames of environmental images; determining the second depth information of the same target object in each selected frame of the environmental image according to preset parameters of a time node corresponding to each frame of the environmental image in the preset number of environmental images; and determining the target depth information of the same target object according to the first depth information and the second depth information.

[0006] In the depth estimation method of the traffic sign of the above embodiment, by acquiring multiple frames of continuous environmental images and determining the position, category and first depth information of the target object in each frame of the environmental image, a data basis is provided for the depth estimation of the subsequent target object. Based on the position and category of the target object, a preset number of environmental images containing the same target object and continuous in time are selected from the multiple frames of continuous environmental images, ensuring the continuity of the tracking process and the consistency of the target object, so that the target object can maintain the continuity of tracking even when it moves quickly or the environment changes. By determining the second depth information of the same target object in each selected frame of the environmental image according to the preset parameters of the time node corresponding to each frame of the environmental image in the preset number of environmental images, the second depth information can take into account various information, and thus can more accurately reflect the actual depth change of the target object, so as to provide a stable depth estimation in complex and changeable scenes. By combining the first depth information and the second depth information of the time-continuous frame to obtain the depth information of the target object, the uncertainty of the depth estimation of a single frame image is reduced, and the optimization of the depth estimation is achieved. Based on this, the present application improves the depth estimation accuracy of the traffic sign, thereby improving the positioning accuracy of the traffic sign. In some embodiments of the present application, the position of the target object includes the pixel coordinates of the target object in each frame of the environmental image, the preset parameters include the navigation positioning information of the vehicle, the relative relationship between the camera on the vehicle for collecting the environmental image and the vehicle, and the internal parameters of the camera, and the second depth information of the same target object in each selected frame of the environmental image is determined according to the preset parameters of the time node corresponding to each frame of the environmental image in the preset number of environmental images, including: determining the first pose conversion parameter from the vehicle coordinate system to the world coordinate system based on the navigation positioning information of the vehicle; determining the second pose conversion parameter from the camera coordinate system to the vehicle coordinate system based on the relative relationship between the camera and the vehicle; determining the second depth information of the same target object in each selected frame of the environmental image according to the first pose conversion parameter, the second pose conversion relationship, the internal parameters of the camera, and the pixel coordinates of the same target object.

[0007] In some embodiments of the present application, determining the second depth information of the same target object in each selected frame of the environmental image based on the first pose conversion parameter, the second pose conversion relationship, the internal parameters of the camera, and the pixel coordinates includes: constructing a first linear equation based on the first pose conversion parameter, the first linear equation is related to the position of the same target object in the vehicle coordinate system at different time nodes; constructing a second linear equation based on the second pose conversion parameter, the second linear equation is related to the position and second depth information of the same target object in the vehicle coordinate system at different time nodes; solving the overdetermined set of equations formed by the first linear equation and the second linear equation of the same target object based on the least squares method, to determine the second depth information of the same target object in each selected frame of the environmental image.

[0008] In some embodiments of the present application, determining the position, category and first depth information of the target object in each frame of the environmental image includes: determining the position, category and first depth information of the target object in each frame of the environmental image based on a trained neural network model.

[0009] In some embodiments of the present application, the training method of the neural network model includes: obtaining a training data set, the training data set including multiple historical images and a label for each historical image, the label including a location label, a category label, and a depth image corresponding to each historical image of the target object; based on the location label and category label of the target object in each historical image, performing supervised training on a target detection module of the neural network model; based on the depth image corresponding to each historical image, performing supervised training on a depth prediction module of the neural network model.

[0010] In some embodiments of the present application, based on the position and the category of the target object, a preset number of environmental images that contain the same target object and are temporally continuous are selected from the multiple frames of continuous environmental images, including: based on the trained preset neural network model, an object greater than a preset category confidence threshold in the i-th frame of the environmental image is determined as a first object, and a first category and a first position of the first object are determined, wherein i represents an integer greater than or equal to 1; if a second object of the first category exists in the i+1-th frame of the environmental image, and the distance between the second position of the second object and the first position is less than or equal to a preset distance threshold, the second object is confirmed as the same target object as the first object, and an association relationship of the same target object is established between the i-th frame of the environmental image and the i+1-th frame of the environmental image; an association relationship of a target object, selecting a preset number of environmental images that contain the same target object and are temporally continuous from the multiple frames of continuous environmental images.

[0011] In some embodiments of the present application, the method also includes: if the second object of the first category does not exist in the i+1th frame environmental image, or the distance between the second position of the second object and the first position is greater than the preset distance threshold, the second object is confirmed as not being the same target object as the first object, and no association relationship of the same target object is established between the i-th frame environmental image and the i+1-th frame environmental image.

[0012] In some embodiments of the present application, the method further includes: if the i-th frame environment image and the i+1-th frame environment image have an association relationship with the same target object, and the i+1-th frame environment image and the i+2-th frame environment image do not have an association relationship with the same target object, confirming that the i-th frame environment image and the i+2-th frame environment image do not have an association relationship with the same target object.

[0013] In some embodiments of the present application, the method also includes: determining the number of images in the continuous environmental images that have an association relationship with the same target object; if the number is less than the preset number, obtaining an updated environmental image that has temporal continuity with the multiple frames of continuous environmental images, and obtaining an updated multiple frames of continuous environmental images based on the multiple frames of continuous environmental images and the updated environmental images; determining the position and category of the target object in each frame of the updated multiple frames of continuous environmental images; and based on the position and category of the target object, selecting a preset number of environmental images that contain the same target object and are temporally continuous from the updated multiple frames of continuous environmental images.

[0014] In some embodiments of the present application, determining the target depth of the same target object based on the first depth information and the second depth information includes: determining the weight of the first depth information and the weight of the second depth information, and determining the target depth information based on the weighted sum of the first depth information and the second depth information.

[0015] In a second aspect, the present application also provides a depth estimation device for traffic signs, the device comprising: an acquisition module for acquiring multiple frames of continuous environmental images and determining the position, category and first depth information of the target object in each frame of the environmental image; a selection module for selecting a preset number of environmental images that contain the same target object and are temporally continuous from the multiple frames of continuous environmental images based on the position and category of the target object; a first determination module for determining the second depth information of the same target object in each selected frame of the environmental image based on preset parameters of a time node corresponding to each frame of the environmental image in the preset number of environmental images; and a second determination module for determining the target depth information of the same target object based on the first depth information and the second depth information.

[0016] In a third aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the depth estimation method for traffic signs described in the above embodiment.

[0017] In a fourth aspect, the present application also provides a vehicle, which includes the electronic device described in the above embodiment.

[0018] It can be understood that the depth estimation device of the traffic sign of the second aspect, the electronic device of the third aspect and the vehicle of the fourth aspect provided above all correspond to the depth estimation method of the traffic sign of the first aspect. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding depth estimation method of traffic signs provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of an application scenario of a depth estimation method for traffic signs provided in one embodiment of the present application.

[0020] Figure 2 It is a flowchart of a depth estimation method for a traffic sign provided in an embodiment of the present application.

[0021] Figure 3 It is a flowchart of a training method for a neural network model provided in one embodiment of the present application.

[0022] Figure 4 It is a detailed flow chart of step S11 in the depth estimation method of a traffic sign provided in an embodiment of the present application.

[0023] Figure 5 It is a detailed flow chart of step S12 in the depth estimation method of a traffic sign provided in an embodiment of the present application.

[0024] Figure 6 It is a schematic diagram of functional modules of a depth estimation device for a traffic sign provided in one embodiment of the present application.

[0025] Component Symbols Vehicle 1 Electronic devices 10 Memory 11 Processor 12 Camera 20 Traffic sign depth estimation device 100 Acquisition module 110 Select Module 120 The first determination module 130 The second determination module 140 The following specific implementation methods will further illustrate the present application in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0026] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0027] In the embodiments of the present application, it should be noted that, unless otherwise clearly specified and limited, words such as "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" is intended to present related concepts in a specific way.

[0028] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or mutual communication; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0029] In the description of the present application, it should be noted that, unless otherwise expressly specified and limited, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In addition, in the description of the present application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically limited.

[0030] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0031] See also Figure 1 , which is a schematic diagram of an application scenario of a depth estimation method for a traffic sign provided in an embodiment of the present application.

[0032] The embodiment of the present application provides a depth estimation method for a traffic sign, which can be applied to one or more electronic devices 10. The electronic device 10 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA), a digital processor (Digital Signal Processor, DSP), an embedded device, etc.

[0033] Specifically, the electronic device 10 is used to acquire multiple frames of continuous environmental images, and determine the position, category and first depth information of the target object in each frame of the environmental image; based on the position and category of the target object, select a preset number of environmental images that contain the same target object and are continuous in time from the multiple frames of continuous environmental images; determine the second depth information of the same target object in each selected frame of the environmental image according to preset parameters of the time node corresponding to each frame of the preset number of environmental images; determine the target depth information of the same target object according to the first depth information and the second depth information.

[0034] In some embodiments of the present application, the electronic device 10 can be communicatively connected to a desktop computer, a notebook, a PDA, a cloud server, or other devices.

[0035] In some embodiments of the present application, the electronic device 10 can perform human-computer interaction with the user through a keyboard, a mouse, a remote control, a touch pad, or a voice control device.

[0036] In some embodiments of the present application, the electronic device 10 may also include a network device and / or a client device. The network device includes but is not limited to a single network server, a server group consisting of multiple network servers, and a cloud server consisting of a large number of hosts or network servers based on cloud computing.

[0037] In some embodiments of the present application, the network where the electronic device 10 is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0038] In some embodiments of the present application, the embodiments of the present application further provide a vehicle 1 , wherein the vehicle 1 includes an electronic device 10 .

[0039] In some embodiments of the present application, the electronic device 10 may be an on-board device of the vehicle 1 , such as a body control module (BCM), a vehicle control unit (VCU), etc.

[0040] In some embodiments of the present application, the vehicle 1 may further include a camera 20, which is installed in front of the vehicle 1. The electronic device 10 is in communication connection with the camera 20. The camera 20 is used to collect multiple frames of continuous environmental images around the vehicle 1, and upload the collected multiple frames of continuous environmental images to the electronic device 10.

[0041] See also Figure 2 , is a schematic diagram of the steps of a depth estimation method for a traffic sign provided in an embodiment of the present application.

[0042] Specifically, the depth estimation method of the traffic sign includes the following steps. According to different requirements, the order of some steps in the flowchart can be changed, and some steps can be omitted.

[0043] Step S10: acquiring multiple frames of continuous environment images, and determining the position, category and first depth information of the target object in each frame of the environment image.

[0044] In some embodiments of the present application, multiple frames of continuous environmental images around the vehicle 1 are collected by the camera 20, and the collected multiple frames of continuous environmental images are uploaded to the electronic device 10. The camera 20 may include but is not limited to a periscopic camera, a surround camera, etc. The camera 20 may collect environmental images at a preset collection frame rate, for example, collecting an environmental image every 0.5 seconds.

[0045] In some embodiments of the present application, after acquiring the environmental image, the electronic device 10 may also perform preprocessing on the environmental image, such as image cleaning, image enhancement, image standardization, etc., which can improve image quality, enhance key features, and standardize data distribution, and help improve the accuracy and efficiency of subsequent depth estimation steps.

[0046] In some embodiments of the present application, the position of the target object may include, but is not limited to, the pixel coordinates of the target object in each frame of the environment image, such as multiple vertex coordinates of a bounding box, the center point coordinates of a bounding box, etc. Among them, in the field of image processing, a bounding box is a rectangular area used to represent the position of a target object in an image. The multiple vertex coordinates of a bounding box may include the upper left vertex coordinates, the lower left vertex coordinates, the upper right vertex coordinates, the lower right vertex coordinates, etc. of the bounding box. The center point coordinates of the bounding box refer to the geometric center of the rectangular box used to locate and identify the target object in a two-dimensional image.

[0047] In some embodiments of the present application, the categories of target objects may include but are not limited to traffic signs, such as speed limit signs, no turn signs, stop signs, etc.; traffic lights, red light, yellow light, green light; road surface markings, such as lane lines (solid lines, dashed lines, double yellow lines, etc.), guide arrows, stop lines, etc.

[0048] In some embodiments of the present application, the first depth information of the target object may include but is not limited to the distance between the target object and the vehicle 1 .

[0049] In some embodiments of the present application, the specific steps of determining the position, category, and first depth information of the target object in each frame of the environmental image include: determining the position, category, and first depth information of the target object in each frame of the environmental image based on the trained neural network model. The first depth information may include the average value of all depth values ​​of all pixels corresponding to the target object. For example, the neural network model predicts a predicted bounding box of the target object and the depth value of each pixel in the environmental image, and the pixels in the predicted bounding box can be used as all pixels corresponding to the target object, and the first depth information is determined based on the average value of the depth values ​​of all pixels in the predicted bounding box.

[0050] In the above embodiment, through the application of the trained neural network model, the position, category and first depth of the target object are automatically determined, which improves the processing efficiency and accuracy of determining the position, category and first depth of the target object.

[0051] In some embodiments of the present application, a training method for a neural network model includes: obtaining a training data set, the training data set including multiple historical images and a label for each historical image, the label including a location label, a category label, and a depth image corresponding to each historical image of the target object; based on the location label and category label of the target object in each historical image, performing supervised training on a target detection module of the neural network model; based on the depth image corresponding to each historical image, performing supervised training on a depth prediction module of the neural network model.

[0052] In the above embodiment, by simultaneously training the target detection module and the depth prediction module of the neural network model, the neural network model can simultaneously learn to identify the location and category of the target object, as well as predict the depth information of the target object. Supervised training ensures that the neural network model can learn from labeled historical images, improving the accuracy of target detection and the accuracy of depth prediction. The use of depth images enables the neural network model to learn the depth changes of the target object under different lighting, weather and viewing angles, and improves the robustness of the neural network model in complex environments. By training on a variety of training data sets, the neural network model can learn the characteristics of target objects in different scenarios and conditions, improve the generalization ability of the neural network model on unknown data, and help improve the model's positioning accuracy for target objects, whether in complex traffic scenarios or under different environmental conditions, and can more accurately locate the target object.

[0053] It should be noted that the specific steps of the neural network model training method will be described later. Figure 3 The steps shown are described in detail and will not be repeated here to avoid repetition.

[0054] Step S11 : selecting a preset number of environmental images that contain the same target object and are continuous in time from a plurality of frames of continuous environmental images based on the position and category of the target object.

[0055] Specifically, step S11, based on the position and category of the target object, selects a preset number of environmental images that contain the same target object and are continuous in time from multiple frames of continuous environmental images, specifically including: based on the trained neural network model, determining an object in the i-th frame of the environmental image that is greater than a preset category confidence threshold as the first object, and determining the first category and the first position of the first object, wherein i represents an integer greater than or equal to 1; if there is a second object of the first category in the i+1-th frame of the environmental image, and the distance between the second position of the second object and the first position is less than or equal to the preset distance threshold, confirming the second object as the same target object as the first object, and establishing an association relationship with the same target object between the i-th frame of the environmental image and the i+1-th frame of the environmental image; based on the association relationship with the same target object, selecting a preset number of environmental images that contain the same target object and are continuous in time from the continuous environmental images.

[0056] In the above embodiment, by analyzing the position and category of the object in the continuous environment image, the real-time tracking of the same target object is realized. By setting the category confidence threshold and the distance threshold, the situation of mistracking and target switching is reduced, and the probability of the target object being correctly identified and tracked in the continuous environment image is increased. By establishing the association relationship of the target object between the continuous frames, the stability of the tracking is improved, and the tracking can be maintained even when the target object is temporarily blocked or leaves the field of view, which can adapt to complex dynamic scenes, improve the adaptability to environmental changes, and enhance the robustness.

[0057] It should be noted that the specific steps of how to select a preset number of environmental images containing the same target object and being continuous in time from multiple frames of continuous environmental images based on the position and category of the target object will be described in detail later. Figure 4 The steps shown are described in detail. In order to avoid repetition, the detailed description will not be repeated here. In order to avoid repetition, the detailed description will not be repeated here.

[0058] In some embodiments of the present application, the depth estimation method of traffic signs also includes: if there is no second object of the first category in the i+1th frame of the environmental image, or the distance between the second position of the second object and the first position is greater than a preset distance threshold, the second object is confirmed as not being the same target object as the first object, and no association relationship with the same target object is established between the i-th frame of the environmental image and the i+1-th frame of the environmental image.

[0059] In the above embodiment, it is ensured that the association relationship is established only when the position and category of the target object in consecutive frames match, thereby avoiding erroneous association of different target objects as the same target, reducing misjudgment in the tracking process, and improving the accuracy of target object tracking.

[0060] In some embodiments of the present application, the depth estimation method of traffic signs also includes: if the i-th frame environment image and the i+1-th frame environment image have an association relationship with the same target object, and the i+1-th frame environment image and the i+2-th frame environment image do not have an association relationship with the same target object, confirming that the i-th frame environment image and the i+2-th frame environment image do not have an association relationship with the same target object.

[0061] In the above embodiment, if the target object is associated between the i-th frame and the i+1-th frame, but not between the i+1-th frame and the i+2-th frame, it can be inferred that there is no association between the i-th frame and the i+2-th frame. If the target object is lost in an intermediate frame, the association relationship of the same target object cannot be established across frames, which helps to avoid the propagation of erroneous tracking information, ensures that the tracking of the target object is consistent between consecutive frames, and improves the tracking accuracy.

[0062] In some embodiments of the present application, the depth estimation method of traffic signs also includes: determining the number of images in continuous environmental images that have an association relationship with the same target object; if the number is less than a preset number, obtaining an updated environmental image that has temporal continuity with the multiple frames of continuous environmental images, and obtaining an updated multiple frames of continuous environmental images based on the multiple frames of continuous environmental images and the updated environmental images; determining the position and category of the target object in each frame of the updated multiple frames of continuous environmental images; and selecting a preset number of environmental images that contain the same target object and are temporally continuous from the updated multiple frames of continuous environmental images based on the position and category of the target object.

[0063] Among them, the preset number can be set according to actual needs. For example, the preset number can be an integer greater than or equal to 3, and this application does not impose any specific restrictions on this.

[0064] In the above embodiment, if the number of associated images in the continuous images is less than a preset number, an updated environment image will be acquired to provide richer data support for the subsequent determination of the second depth information of the same target object, which helps to improve the reliability of depth estimation.

[0065] Step S12, determining second depth information of the same target object in each selected frame of the environmental image according to preset parameters of a time node corresponding to each frame of the environmental image in a preset number of environmental images.

[0066] Specifically, the preset parameters include navigation positioning information of vehicle 1, the relative relationship between a camera on vehicle 1 for collecting environmental images and vehicle 1, and the internal parameters of the camera.

[0067] In some embodiments of the present application, step S12 determines the second depth information of the same target object in each selected frame of the environmental image according to preset parameters of the time node corresponding to each frame of the environmental image in a preset number of environmental images, specifically including: determining the first pose conversion parameter from the vehicle coordinate system to the world coordinate system based on the navigation positioning information of the vehicle 1; determining the second pose conversion parameter from the camera coordinate system to the vehicle coordinate system based on the relative relationship between the camera and the vehicle 1; determining the second depth information of the same target object in each selected frame of the environmental image according to the first pose conversion parameter, the second pose conversion relationship, the intrinsic parameters of the camera, and the pixel coordinates of the same target object.

[0068] In the above embodiment, the second depth information of the target object in each frame of the environmental image is determined by combining the vehicle's navigation positioning information, the relative relationship between the camera and the vehicle, the camera's intrinsic parameters, and the pixel coordinates of the target object. The vehicle's navigation positioning information, camera's intrinsic parameters, and image data are integrated to improve the accuracy and robustness of depth estimation.

[0069] It should be noted that how to determine the second depth information of the same target object in each selected frame of the environment image in the subsequent according to the preset parameters of the time node corresponding to each frame of the environment image in the preset number of environment images Figure 5 The steps shown are described in detail and will not be repeated here to avoid repetition.

[0070] Step S13: determining target depth information of the same target object according to the first depth information and the second depth information.

[0071] Specifically, step S13, determining the target depth information of the same target object according to the first depth information and the second depth information specifically includes: determining the weight of the first depth information and the weight of the second depth information, and determining the target depth information based on the weighted sum of the first depth information and the second depth information, and the specific calculation formula is: Wherein, D is the target depth information, Dm is the first depth information, Di is the second depth information, a represents the weight of the first depth information, and (1-a) represents the weight of the second depth information. Wherein, a is a constant, and the value of a can be obtained based on experience, and this application does not impose any specific restrictions on this.

[0072] In the depth estimation method of the traffic sign of the embodiment of the present application, by acquiring multiple continuous environmental images and determining the position, category and first depth information of the target object in each frame of the environmental image, a data basis is provided for the depth estimation of the subsequent target object. Based on the position and category of the target object, a preset number of environmental images containing the same target object and continuous in time are selected from multiple continuous environmental images, ensuring the continuity of the tracking process and the consistency of the target object, so that the target object can maintain the continuity of tracking in the case of rapid movement or environmental changes. By determining the second depth information of the same target object in each selected frame of the environmental image according to the preset parameters of the time node corresponding to each frame of the environmental image in the preset number of environmental images, the second depth information can take into account various information, and thus can more accurately reflect the actual depth change of the target object, so as to provide a stable depth estimation in complex and changeable scenes. By combining the first depth information and the second depth information of the time-continuous frame to obtain the depth information of the target object, the uncertainty of the depth estimation of a single frame image is reduced, and the optimization of the depth estimation is achieved. Based on this, the present application improves the depth estimation accuracy of the traffic sign, thereby improving the positioning accuracy of the traffic sign.

[0073] See also Figure 3 , is a schematic diagram of the specific steps of the training method of the neural network model provided in one embodiment of the present application.

[0074] This embodiment is a specific description of the training method of the neural network model in the above embodiment, and further describes the training method of the neural network model. Specifically, it includes the following steps: Step S101, obtaining a training data set.

[0075] In some embodiments of the present application, the training data set includes multiple historical images (e.g., environment images) and labels for each historical image, and the labels include a location label of the target object, a category label, and a depth image corresponding to each historical image. The location label may be a real bounding box of the area where the target object is located, and the depth image may include the depth value of each pixel in the historical image. The depth value of the pixel in the real bounding box (e.g., the average depth value of all pixels in the real bounding box) may indicate the distance between the target object and the vehicle 1.

[0076] In some embodiments of the present application, in order to achieve more accurate prediction of discrete depth values ​​of an image, the depth value of the depth image can be divided according to a preset range to obtain multiple depth intervals. For example, if the minimum depth value of the depth image is 0 and the maximum depth value is 10 meters, 10 depth intervals can be obtained by dividing the range into 1 meter each.

[0077] Step S102, based on the location label and category label of the target object in each historical image, supervised training is performed on the target detection module of the neural network model.

[0078] In some embodiments of the present application, the neural network model may include but is not limited to a feedforward neural network (FNN) model, a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a deep structured semantic model (DSSM), etc. The model structure of the preset model can be selected according to actual needs, and the present application does not impose specific restrictions on this.

[0079] In some embodiments of the present application, the neural network model includes a target detection module, which can be used to predict the target object in the environment image. For example, the target detection module is used to determine the target object in the environment image and the predicted category corresponding to the target object.

[0080] In one example, the target detection module can be used to determine the predicted category confidence of each object in the environment image, take the object whose predicted category confidence is greater than the preset category confidence threshold as the target object, and determine the predicted category corresponding to the target object. Among them, the target object includes multiple categories, each category corresponds to a category confidence threshold, and the predicted category confidence of each object is compared with the category confidence threshold of each category. If it is greater than the corresponding category confidence threshold, it can be considered as the target object of the corresponding category.

[0081] In one example, the target detection module may also be used to determine the position of each target object, for example, to generate a predicted bounding box for each target object, and to determine the position of the target object based on vertex coordinates or center point coordinates of the predicted bounding box.

[0082] In some embodiments of the present application, the target detection module can be supervised trained based on the position label and category label of the target object in each historical image. The training steps of the target detection module include: inputting the historical images in the training data set into the target detection module in batches, and outputting the predicted position and predicted category of the target object in each historical image through the target detection module. Determine the distance between the predicted position and the position label (for example, the Euclidean distance between the center point of the predicted bounding box and the true bounding box) and the difference between the predicted category and the category label. Based on the distance and the difference, the target detection module is iteratively optimized. Through iterative optimization, the target detection module can gradually improve its accuracy in predicting the position and category of the target object.

[0083] Step S103, based on the depth image corresponding to each historical image, supervised training is performed on the depth prediction module of the neural network model.

[0084] In some embodiments of the present application, the neural network model includes a depth prediction module, which can be used to predict the depth information of the target object in the environmental image. The training steps of the depth prediction module are as follows: the historical images in the training data set are input into the depth prediction module in batches, the depth prediction module extracts the depth features of the historical images, and outputs the depth distribution of each historical image. The depth distribution includes the probability value of each pixel point in the historical image belonging to each depth interval, and the sum of the probabilities of all pixels is 1; after obtaining the depth distribution, the expected distribution of each pixel point in the historical image is calculated, that is, the expected depth value of each pixel point is determined: the probability of each pixel point is multiplied by the median value of the corresponding depth interval, and then all products are added. For example, pixel A has a 30% probability of belonging to the interval of 0 to 1 meter and a 70% probability of belonging to the interval of 1 to 2 meters; the median of the interval of 0 to 1 meter (for example, 0.5 meters) can be multiplied by 30%, and the midpoint of the interval of 1 to 2 meters (for example, 1.5 meters) can be multiplied by 70%, and then the two results can be added together to obtain the expected depth value of pixel A; after determining the expected depth value of each pixel, it can be compared with the depth image in the label of the historical image to determine the difference between the expected depth predicted by the depth prediction module and the actual depth, so as to optimize the depth prediction module when the difference is large. Among them, the corresponding expected depth image can be generated according to the expected depth value of each pixel, and the gray value of each pixel in the expected depth image represents the depth value. By comparing this expected depth image with the depth image in the label, the difference between the expected depth predicted by the depth prediction module and the actual depth can be obtained.

[0085] In some embodiments of the present application, the prediction error of the neural network model can be minimized and the generalization ability of the model can be improved by adjusting the internal parameters and structure of the neural network model based on optimization methods such as gradient descent and back propagation. When the neural network model reaches the preset training conditions on the training data set, it can be considered that the trained neural network model is obtained. For example, the preset training conditions may include but are not limited to the training iteration reaching the preset number of iterations, the model prediction accuracy reaching the preset accuracy threshold, etc.

[0086] In the above implementation, by simultaneously training the target detection module and the depth prediction module of the neural network model, the neural network model can simultaneously learn to identify the location and category of the target object, as well as predict the depth information of the target object. Supervised training ensures that the neural network model can learn from labeled historical images, improving the accuracy of target detection and the accuracy of depth prediction. The use of depth images enables the neural network model to learn the depth changes of the target object under different lighting, weather and viewing angles, and improves the robustness of the neural network model in complex environments. By training on a variety of training data sets, the neural network model can learn the characteristics of target objects in different scenarios and conditions, improve the generalization ability of the neural network model on unknown data, and help improve the model's positioning accuracy for target objects, whether in complex traffic scenarios or under different environmental conditions, and can more accurately locate the target object.

[0087] See also Figure 4 , is a schematic diagram of specific steps of step S11 in the depth estimation method of traffic signs provided in one embodiment of the present application.

[0088] This embodiment is a specific description of step S11 in the above embodiment, and further describes how to select a preset number of environmental images containing the same target object and being continuous in time from multiple frames of continuous environmental images based on the position and category of the target object. Specifically, the following steps are included: Step S111, based on the trained neural network model, determining an object in the i-th frame environment image that is greater than a preset category confidence threshold as a first object, and determining a first category and a first position of the first object.

[0089] Here, i represents an integer greater than or equal to 1. For example, the i-th frame of the environment image represents the i-th image in the chronological order of multiple frames of continuous environment images.

[0090] In some embodiments of the present application, the neural network module includes a target detection module. The target detection module can be used to determine the predicted category confidence of each object in the environmental image, and take the object whose predicted category confidence in the i-th frame environmental image is greater than the preset category confidence threshold as the first object, and determine the first category corresponding to the first object. Among them, the object includes multiple categories, each category corresponds to a category confidence threshold, and the predicted category confidence of each object is compared with the category confidence threshold of each category. If it is greater than the corresponding category confidence threshold, it can be considered as the first object of the corresponding category. The target detection module can also be used to determine the position of each object, for example, generate a predicted bounding box for each first object, and determine the first position corresponding to the first object based on the vertex coordinates or center point coordinates of the predicted bounding box.

[0091] In some embodiments of the present application, the detection of the target detection module can be reduced to ensure the recall of the object, so that more detection results are output to the tracking.

[0092] Step S112: If there is a second object of the first category in the i+1th frame environmental image, and the distance between the second position of the second object and the first position is less than or equal to a preset distance threshold, the second object is confirmed as the same target object as the first object, and an association relationship of the same target object is established between the i-th frame environmental image and the i+1-th frame environmental image.

[0093] In some embodiments of the present application, the distance between the second position and the first position can be determined by calculating a cost matrix, wherein the cost matrix is ​​used to evaluate the degree of association between the detection result and the known trajectory. The specific calculation formula is as follows: If the Intersection over Union (IoU) is greater than 0, it means that there is an overlap between the first predicted bounding box of the first object in the i-th frame environment image and the second predicted bounding box of the second object in the i+1-th frame environment image. At this time, the cost can be determined based on the IoU value, because the higher the IoU value, the greater the overlap between the first predicted bounding box and the second predicted bounding box, that is, the smaller the distance between the first predicted bounding box and the second predicted bounding box, the greater the possibility of the first object being associated with the second object. The IoU value calculation formula is the ratio of the intersection area of ​​the first predicted bounding box and the union area of ​​the second predicted bounding box.

[0094] If IoU is equal to 0, it means that the first predicted bounding box and the second predicted bounding box do not overlap. At this time, the cost can be set to the Euclidean distance from the minimum vertex. The Euclidean distance is the straight-line distance between two points. Here it refers to the distance between the minimum vertices of the first predicted bounding box and the second predicted bounding box. The smaller the distance, the closer the first predicted bounding box and the second predicted bounding box are, and the greater the possibility of association.

[0095] Step S113 : Based on the association relationship of the same target object, a preset number of environmental images that contain the same target object and are continuous in time are selected from a plurality of frames of continuous environmental images.

[0096] In some embodiments of the present application, the preset number can be set according to actual needs. For example, the preset number can be an integer greater than or equal to 3, and the present application does not impose any specific restrictions on this.

[0097] In the above embodiment, by analyzing the position and category of the object in the continuous environment image, the real-time tracking of the same target object is achieved. By setting the category confidence threshold and the distance threshold, the situation of mistracking and target switching is reduced, and the probability of the target object being correctly identified and tracked in the continuous environment image is increased. By establishing the association relationship of the target object between the continuous frames, the stability of the tracking is improved, and even if the target object is temporarily blocked or leaves the field of view, it can still be tracked in other continuous frames, which can adapt to complex dynamic scenes, improve the adaptability to environmental changes, and enhance robustness.

[0098] See also Figure 5 , is a schematic diagram of specific steps of step S12 in the depth estimation method of traffic signs provided in one embodiment of the present application.

[0099] This embodiment is a specific description of step S12 in the above embodiment, and further describes how to determine the second depth information of the same target object in each selected frame of the environment image according to the preset parameters of the time node corresponding to each frame of the environment image in the preset number of environment images. Specifically, the following steps are included: Step S121, based on the navigation positioning information of the vehicle, determine the first position conversion parameter from the vehicle coordinate system to the world coordinate system.

[0100] In some embodiments of the present application, the navigation positioning information of the vehicle 1 is obtained by a navigation system installed in the vehicle 1, for example, the navigation system includes but is not limited to the Global Navigation Satellite System 110 (GNSS) and the Inertial Measurement Unit 120 (IMU). In the navigation positioning information, the first position conversion parameter from the vehicle coordinate system to the world coordinate system is known.

[0101] Step S122, based on the relative relationship between the camera and the vehicle, determine a second pose conversion parameter from the camera coordinate system to the vehicle coordinate system.

[0102] In some embodiments of the present application, the relative relationship between the camera 20 and the vehicle 1 is known. For example, the relative relationship between the camera 20 and the vehicle 1 may include but is not limited to the installation position of the camera 20 in the vehicle coordinate system, the installation angle of the camera 20, etc., and further, the second posture conversion parameters from the camera coordinate system to the vehicle coordinate system may be determined according to the relative relationship between the camera 20 and the vehicle 1. In one example, the second posture conversion parameters are known.

[0103] Step S123, determining the second depth information of the same target object in each selected frame of the environment image according to the first pose conversion parameter, the second pose conversion relationship, the internal parameter of the camera, and the pixel coordinates of the same target object.

[0104] In some embodiments of the present application, the calculation method for determining the second depth information of the same target object in each selected frame of the environment image according to the first pose conversion parameter, the second pose conversion relationship, the camera's internal parameters, and the pixel coordinates specifically includes: 1. Based on the first pose transformation parameter, a first linear equation is constructed, which is related to the position of the same target object in the vehicle coordinate system at different time nodes.

[0105] Assume that the position of the target object in the vehicle coordinate system at time t1 is , the position of the target object in the vehicle coordinate system at time t2 is , where time node t2 represents the adjacent time node after time node t1.

[0106] Assume that the first pose conversion parameters from the vehicle coordinate system to the world coordinate system at time t1 are R1 and T1, the first pose conversion parameters from the vehicle coordinate system to the world coordinate system at time t2 are R2 and T2, and the relative pose conversion parameters from the vehicle coordinate system to the world coordinate system at time t1 and time t2 are and ,in: Expanding the following formula, we can get 3 answers about The first linear equation of .

[0107] 2. Based on the second posture transformation parameter, a second linear equation is constructed, where the second linear equation is related to the position and second depth information of the same target object in the vehicle coordinate system at different time nodes.

[0108] Assume the second pose transformation parameter from the camera coordinate system to the vehicle coordinate system is and , the camera’s internal parameters are , the pixel coordinates of the target object at time t1 are , the pixel coordinates at time t1 are , the second depth information of the target object at time t1 is , the second depth information of the target object at time t1 is .

[0109] Expanding the following formula, we can get 3 answers about The second linear equation of .

[0110] Expanding the following formula, we can get 3 answers about The second linear equation of .

[0111] 3. Solve the overdetermined equation group formed by the first linear equation and the second linear equation of the same target object based on the least square method to determine the second depth information of the same target object in each selected frame of the environment image.

[0112] From this, a total of 9 linear equations can be constructed to solve 8 unknowns , forming an overdetermined system of equations (the number of equations is greater than the number of unknowns). At this time, the least squares method is used to solve this overdetermined system of linear equations.

[0113] In the above embodiment, the second depth information of the target object in each frame of the environmental image is determined by combining the vehicle's navigation positioning information, the relative relationship between the camera and the vehicle, the camera's intrinsic parameters, and the pixel coordinates of the target object. The vehicle's navigation positioning information, camera's intrinsic parameters, and image data are integrated to improve the accuracy and robustness of depth estimation.

[0114] See also Figure 6 , is a schematic diagram of functional modules of a depth estimation device 100 for a traffic sign provided in an embodiment of the present application.

[0115] In this embodiment, based on the above Figure 2 The present application also provides a traffic sign depth estimation method 100 with the same concept as the traffic sign depth estimation method in the illustrated embodiment, which can be used to perform the above-mentioned traffic sign depth estimation method. For ease of explanation, the schematic diagram of the composition of the traffic sign depth estimation method 100 embodiment only shows the parts related to the embodiment of the present application. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the traffic sign depth estimation method 100, which can include more or fewer components than shown in the illustration, or combine certain components, or arrange the components differently.

[0116] Specifically, the depth estimation device 100 for a traffic sign provided in the embodiment of the present application includes an acquisition module 110 , a selection module 120 , a first determination module 130 , and a second determination module 140 .

[0117] The acquisition module 110 is used to acquire multiple frames of continuous environment images and determine the position, category and first depth information of the target object in each frame of the environment image.

[0118] The selection module 120 is used to select a preset number of environmental images that contain the same target object and are continuous in time from a plurality of frames of continuous environmental images based on the location and category of the target object.

[0119] The first determination module 130 is used to determine the second depth information of the same target object in each selected frame of the environmental image according to preset parameters of a time node corresponding to each frame of the environmental image in a preset number of environmental images.

[0120] The second determining module 140 is configured to determine target depth information of the same target object according to the first depth information and the second depth information.

[0121] In the depth estimation device 100 of the traffic sign of the above embodiment, by acquiring multiple frames of continuous environmental images and determining the position, category and first depth information of the target object in each frame of the environmental image, a data basis is provided for the subsequent depth estimation of the target object. Based on the position and category of the target object, a preset number of environmental images containing the same target object and continuous in time are selected from multiple frames of continuous environmental images, ensuring the continuity of the tracking process and the consistency of the target object, so that the target object can maintain the continuity of tracking even when it moves quickly or the environment changes. By determining the second depth information of the same target object in each selected frame of the environmental image according to the preset parameters of the time node corresponding to each frame of the environmental image in the preset number of environmental images, the second depth information can take into account various information, and thus can more accurately reflect the actual depth change of the target object, so as to provide a stable depth estimation in complex and changeable scenes. By combining the first depth information and the second depth information of the time-continuous frame to obtain the depth information of the target object, the uncertainty of the depth estimation of a single frame image is reduced, and the optimization of the depth estimation is achieved. Based on this, the present application improves the depth estimation accuracy of the traffic sign, thereby improving the positioning accuracy of the traffic sign. Combination Figure 1 As shown, in some embodiments of the present application, the electronic device 10 includes, but is not limited to, a memory 11, a processor 12, and a computer program stored in the memory 11 and executable on the processor 12, such as a depth estimation program for traffic signs. When the computer program is executed by the processor, a depth estimation method for traffic signs as in the above-mentioned embodiment is implemented.

[0122] Figure 1 Only the electronic device 10 having the memory 11 and the processor 12 is shown, and those skilled in the art can understand that Figure 1 The structure shown does not constitute a limitation on the electronic device 10 , and the electronic device 10 may include fewer or more components than shown in the figure, or combine some components, or arrange the components differently.

[0123] The memory 11 in the electronic device 10 stores multiple computer-readable instructions to implement a depth estimation method for traffic signs, and the processor 12 can execute the multiple instructions to achieve: obtaining multiple frames of continuous environmental images, and determining the position, category and first depth of the target object in each frame of the environmental image; based on the position and category of the target object, selecting a preset number of environmental images that contain the same target object and are continuous in time from the continuous environmental images; determining the second depth of the same target object in each selected frame of the environmental image according to preset parameters of the time node corresponding to each frame of the preset number of environmental images; determining the target depth of the same target object according to the first depth and the second depth.

[0124] Specifically, the specific implementation method of the processor 12 for the above instructions can refer to Figure 2 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0125] Those skilled in the art will appreciate that the schematic diagram is merely an example of the electronic device 10 and does not constitute a limitation on the electronic device 10. The electronic device 10 may be a bus-type structure or a star-type structure. The electronic device 10 may also include more or less other hardware or software than shown in the figure, or a different arrangement of components. For example, the electronic device 10 may also include input and output devices, network access devices, etc.

[0126] It should be noted that the electronic device 10 is only an example, and other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and included here by reference.

[0127] Among them, the memory 11 includes at least one type of computer-readable storage medium, and the computer-readable storage medium can be non-volatile or volatile. Computer-readable storage media include flash memory, mobile hard disk, multimedia card, card-type memory (such as SD memory, DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 10, such as a mobile hard disk of the electronic device 10. In other embodiments, the memory 11 can also be an external storage device of the electronic device 10, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 10. The memory 11 can not only be used to store application software and various types of data installed in the electronic device 10, such as the code of a depth estimation program of a traffic sign, etc., but also can be used to temporarily store data that has been output or is to be output.

[0128] In some embodiments, the processor 12 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 12 is the control core (Control Unit) of the electronic device 10, and uses various interfaces and lines to connect various components of the entire electronic device 10, and executes various functions and processes data of the electronic device 10 by running or executing programs or modules stored in the memory 11 (for example, executing a depth estimation program for a traffic sign, etc.), and calling data stored in the memory 11.

[0129] The processor 12 executes the operating system of the electronic device 10 and various installed applications. The processor 12 executes the application to implement the steps in the above-mentioned embodiments of the method for estimating the depth of a traffic sign, for example Figure 2-Figure 5 Steps shown.

[0130] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 11, and executed by the processor 12 to complete the present application. One or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 10. For example, the computer program may be divided into an acquisition module 110, a selection module 120, a first determination module 130, and a second determination module 140.

[0131] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, and includes a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute a part of a depth estimation method for a traffic sign in each embodiment of the present application.

[0132] If the module / unit integrated in the electronic device 10 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware devices to complete through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0133] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. Computer readable media may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory and other memory, etc.

[0134] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0135] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. Figure 1 Only one arrow is used in the figure, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 11 and at least one processor 12, etc.

[0136] An embodiment of the present application also provides a computer-readable storage medium (not shown), in which computer-readable storage medium is stored computer-readable instructions, which are executed by a processor in the electronic device 10 to implement a depth estimation method for a traffic sign in any of the above embodiments.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation.

[0138] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0140] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A method for estimating the depth of a traffic sign, characterized in that: The method comprises: Acquire multiple frames of continuous environment images, and determine the position, category and first depth information of the target object in each frame of the environment image; Based on the position and the category of the target object, selecting a preset number of environmental images that contain the same target object and are continuous in time from the multiple frames of continuous environmental images; Determining second depth information of the same target object in each selected frame of the environmental image according to preset parameters of a time node corresponding to each frame of the environmental image in the preset number of environmental images; The target depth information of the same target object is determined according to the first depth information and the second depth information.

2. The method for estimating the depth of a traffic sign according to claim 1, wherein: The position of the target object includes pixel coordinates of the target object in each frame of the environmental image, the preset parameters include navigation positioning information of the vehicle, a relative relationship between a camera on the vehicle for collecting the environmental image and the vehicle, and an internal parameter of the camera, and determining the second depth information of the same target object in each selected frame of the environmental image according to the preset parameters of the time node corresponding to each frame of the environmental image in the preset number of environmental images includes: Based on the navigation positioning information of the vehicle, determining a first pose conversion parameter from a vehicle coordinate system to a world coordinate system; Based on the relative relationship between the camera and the vehicle, determining a second pose conversion parameter from the camera coordinate system to the vehicle coordinate system; Determine second depth information of the same target object in each selected frame of the environment image according to the first posture conversion parameter, the second posture conversion relationship, the intrinsic parameter of the camera, and the pixel coordinates of the same target object.

3. The method for estimating the depth of a traffic sign according to claim 2, wherein: The determining, according to the first posture conversion parameter, the second posture conversion relationship, the internal parameter of the camera, and the pixel coordinates, the second depth information of the same target object in each selected frame of the environment image includes: Based on the first posture transformation parameter, construct a first linear equation, wherein the first linear equation is related to the position of the same target object in the vehicle coordinate system at different time nodes; Based on the second posture conversion parameter, construct a second linear equation, where the second linear equation is related to the position and second depth information of the same target object at different time nodes in the vehicle coordinate system; An overdetermined system of equations formed by the first linear equation and the second linear equation of the same target object is solved based on the least squares method to determine second depth information of the same target object in each selected frame of the environment image.

4. The method for estimating the depth of a traffic sign according to claim 1, wherein: The determining of the position, category and first depth information of the target object in each frame of the environment image includes: Based on the trained neural network model, the position, category and first depth information of the target object in each frame of the environment image are determined.

5. The method for estimating the depth of a traffic sign according to claim 4, characterized in that: The training method of the neural network model includes: Acquire a training data set, wherein the training data set includes a plurality of historical images and a label of each historical image, wherein the label includes a location label and a category label of a target object and a depth image corresponding to each historical image; Based on the location label and category label of the target object in each historical image, the target detection module of the neural network model is supervisedly trained; Based on the depth image corresponding to each historical image, supervised training is performed on the depth prediction module of the neural network model.

6. The method for estimating the depth of a traffic sign according to claim 4, wherein: The selecting, based on the position and the category of the target object, a preset number of environmental images that contain the same target object and are continuous in time from the multiple frames of continuous environmental images comprises: Based on the trained neural network model, determine an object in the i-th frame of the environmental image that is greater than a preset category confidence threshold as a first object, and determine a first category and a first position of the first object, where i represents an integer greater than or equal to 1; If there is a second object of the first category in the i+1th frame of the environmental image, and the distance between the second position of the second object and the first position is less than or equal to a preset distance threshold, the second object is confirmed as the same target object as the first object, and an association relationship of the same target object is established between the i-th frame of the environmental image and the i+1-th frame of the environmental image; Based on the association relationship of the same target object, a preset number of environmental images that contain the same target object and are continuous in time are selected from the multiple frames of continuous environmental images.

7. The method for estimating the depth of a traffic sign according to claim 6, wherein: The method further comprises: If the second object of the first category does not exist in the i+1th frame environmental image, or the distance between the second position of the second object and the first position is greater than the preset distance threshold, the second object is confirmed as not being the same target object as the first object, and no association relationship of the same target object is established between the i-th frame environmental image and the i+1-th frame environmental image.

8. The method for estimating the depth of a traffic sign according to claim 6, wherein: The method further comprises: If the i-th frame environment image and the i+1-th frame environment image have an association relationship with the same target object, and the i+1-th frame environment image and the i+2-th frame environment image do not have an association relationship with the same target object, confirm that the i-th frame environment image and the i+2-th frame environment image do not have an association relationship with the same target object.

9. The method for estimating the depth of a traffic sign according to claim 6, wherein: The method further comprises: Determine the number of images in the continuous environment images that have an association relationship with the same target object; If the number is less than the preset number, acquiring an updated environmental image having temporal continuity with the multiple frames of continuous environmental images, and obtaining an updated multiple frames of continuous environmental images based on the multiple frames of continuous environmental images and the updated environmental images; Determine the position and category of the target object in each frame of the updated multiple frames of continuous environmental images; Based on the position and the category of the target object, a preset number of environmental images that contain the same target object and are continuous in time are selected from the updated multiple frames of continuous environmental images.

10. The method for estimating the depth of a traffic sign according to claim 1, wherein: The determining the target depth of the same target object according to the first depth information and the second depth information includes: A weight of the first depth information and a weight of the second depth information are determined, and the target depth information is determined based on a weighted sum of the first depth information and the second depth information.

11. A depth estimation device for a traffic sign, characterized in that: The device comprises: An acquisition module, used to acquire multiple frames of continuous environment images and determine the position, category and first depth information of the target object in each frame of the environment image; A selection module, configured to select a preset number of environmental images that contain the same target object and are continuous in time from the multiple frames of continuous environmental images based on the position and the category of the target object; A first determination module, configured to determine the second depth information of the same target object in each selected frame of the environmental image according to preset parameters of a time node corresponding to each frame of the environmental image in the preset number of environmental images; The second determining module is configured to determine target depth information of the same target object according to the first depth information and the second depth information.

12. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the depth estimation method for a traffic sign as described in any one of claims 1 to 10 is implemented.

13. A vehicle, characterized in that: The vehicle includes the electronic device as claimed in claim 12.

Images