Traffic sign detection method, device, storage medium, and electronic apparatus

Abstract

To provide a traffic sign detection method in which a detection error in a traffic sign can be reduced to ensure accuracy of a decision result of a vehicle in order to solve a problem that a detection error in a traffic sign is significant, whereby safety of vehicle driving is improved.SOLUTION: A traffic sign detection method includes a step of determining a traffic sign image frame sequence collected by an on-vehicle camera and odometer frame information corresponding to each frame image in the image frame sequence, a step of determining a residual function and an optimization model based on each frame image and the odometer frame information corresponding to each frame image, a step of determining a target position of the traffic sign in a preset coordinate system based on the residual function and the optimization model, and a step of determining a target position attitude of the traffic sign and a real-time distance between the on-vehicle camera and the traffic sign based on the target position.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to the technical field of intelligent driving, and in particular to a traffic sign detection method, device, computer-readable storage medium and electronic device. [Background technology]

[0002] With the rapid development of social economy, the number of automobiles is increasing day by day, and the effect of the development of intelligent driving technology on the safe driving of automobiles is becoming more and more important. Intelligent driving systems mainly use on-board cameras and on-board radars to sense the environment around the vehicle. The intelligent driving systems in the related art have a large detection error for traffic signs on the road, which affects the safe driving of vehicles. Summary of the Invention [Problem to be solved by the invention]

[0003] In order to solve the above-mentioned problem of large traffic sign detection errors, the present disclosure provides a traffic sign detection method, device, computer-readable storage medium, and electronic device that can reduce traffic sign detection errors and ensure the accuracy of vehicle decision-making results, thereby improving the safety of vehicle driving. [Means for solving the problem]

[0004] A traffic sign detection method according to a first aspect of the present disclosure includes the steps of determining an image frame sequence of a traffic sign collected by an on-board camera of a vehicle and odometer frame information corresponding to each frame image in the image frame sequence, determining a residual function and an optimization model based on each frame image and the odometer frame information corresponding to each frame image, determining a target position of the traffic sign in a predetermined coordinate system based on the residual function and the optimization model, and determining a target position and attitude of the traffic sign or a real-time distance between the on-board camera and the traffic sign based on the target position.

[0005] A traffic sign detection device according to a second aspect of the present disclosure includes a first determination module used to determine an image frame sequence of a traffic sign collected by an on-board camera of a vehicle and odometer frame information corresponding to each frame image in the image frame sequence, a second determination module used to determine a residual function and an optimization model based on each frame image and the odometer frame information corresponding to each frame image, a third determination module used to determine a target position of the traffic sign in a predetermined coordinate system based on the residual function and the optimization model, and a fourth determination module used to determine a target position and attitude of the traffic sign based on the target position.

[0006] A computer-readable storage medium according to a third aspect of the present disclosure has a computer program stored therein, the computer program being used to execute the traffic sign detection method according to the first aspect of the present disclosure.

[0007] An electronic device according to a fourth aspect of the present disclosure includes a processor and a memory for storing instructions executable by the processor, the processor being used to read the instructions from the memory and execute the traffic sign detection method according to the first aspect of the present disclosure.

[0008] A fifth aspect of the present disclosure provides a computer program product, the instructions of which, when executed by a processor, perform the method for detecting traffic signs according to the first aspect of the present disclosure. Effect of the Invention

[0009] The traffic sign detection method disclosed herein is applied to an intelligent driving system, and during vehicle driving, the detection device predicts a target position of the traffic sign in a preset coordinate system based on multi-frame images of the traffic sign collected by an on-board camera and odometer frame information corresponding to the multi-frame images, and further determines a target position and attitude of the traffic sign or a real-time distance between the on-board camera and the traffic sign based on the target position, so that the method can reduce the detection error of the position and attitude and real-time distance of the traffic sign, ensure the accuracy of the vehicle decision-making results, and thereby improve the safety of the vehicle driving. [Brief description of the drawings]

[0010] [Figure 1A] 1 is a schematic diagram of a configuration of a traffic sign detection system according to one exemplary embodiment of the present disclosure. [Figure 1B] FIG. 2 is a schematic diagram of a distance measurement scenario by a distance measurement system according to one exemplary embodiment of the present disclosure. [Figure 1C] FIG. 2 is a schematic diagram of one frame of a traffic light image collected by an in-vehicle camera according to one exemplary embodiment of the present disclosure. [Diagram 2] 2 is a flowchart of a method for detecting traffic signs according to one exemplary embodiment of the present disclosure. [Figure 3A] 1 is a schematic diagram of one frame of a traffic sign board image collected by an in-vehicle camera according to one exemplary embodiment of the present disclosure. FIG. [Figure 3B] 3B is a schematic diagram of a plurality of detection points determined in one frame of the traffic sign board image shown in FIG. 3A. [Figure 4] 5 is a flowchart of a method for detecting traffic signs according to another exemplary embodiment of the present disclosure. [Diagram 5] 4 is a flowchart of a method for detecting the position and orientation of a traffic sign according to an exemplary embodiment of the present disclosure. [Figure 6] 11 is a flowchart of a method for detecting a position and orientation of a traffic sign according to another exemplary embodiment of the present disclosure. [Figure 7]5 is a flowchart of a method for detecting traffic signs according to a further exemplary embodiment of the present disclosure. [Figure 8] 5 is a flowchart of a method for detecting traffic signs according to a further exemplary embodiment of the present disclosure. [Figure 9] 5 is a flowchart of a method for detecting traffic signs according to a further exemplary embodiment of the present disclosure. [Figure 10] 1 is a schematic diagram of a configuration of a traffic sign detection device according to one exemplary embodiment of the present disclosure; [Figure 11] FIG. 13 is a schematic diagram of a configuration of a traffic sign detection device according to another exemplary embodiment of the present disclosure. [Figure 12] FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] In the following, in order to explain the present disclosure, exemplary embodiments of the present disclosure will be described in detail with reference to the drawings. Obviously, it should be understood that the described embodiments are merely a part of the embodiments of the present disclosure, and are not all of the embodiments of the present disclosure, and the present disclosure is not limited to the exemplary embodiments.

[0012] Unless otherwise specified, the relative arrangement of components and steps, formulas and numerical values ​​described in these examples do not limit the scope of the present disclosure. Summary of the Invention

[0013] An intelligent drive system realizes automatic or semi-automatic driving of a vehicle by sensing and analyzing environmental information around the vehicle and autonomously making decisions and controlling the vehicle. An intelligent drive system mainly uses an on-board camera and an on-board radar to collect environmental information around an intelligent drive vehicle (hereinafter simply referred to as a vehicle).

[0014] Accurate recognition of traffic signs is important for the safe driving of intelligent drive vehicles. Road traffic environments are complex and diverse, and in actual driving environments, the image quality of traffic signs collected by on-board cameras may be affected by light irradiation, weather changes, etc. In particular, when traffic signs are tilted or distorted, the accuracy of traffic sign recognition algorithms is greatly reduced. Intelligent drive systems make decisions based on recognition results with large errors, which affects the accuracy of decision-making results and further affects the safety of vehicle driving.

[0015] In the scenario of distance perception, the intelligent driving system obtains the environmental images around the vehicle collected in real time by the on-board camera, predicts the distance between the on-board camera and the traffic sign based on the images collected in real time, and determines and executes the corresponding planned control operation based on the predicted distance. At present, the on-board cameras applied in the field of intelligent driving are mainly monocular cameras or binocular cameras. Distance measurement by monocular cameras is mainly realized based on target recognition and pinhole phenomenon, and has the advantages of simple structure, mature algorithm, small calculation amount, low cost, etc., but is sensitive to weather, light irradiation, etc., and in poor weather and light irradiation environments, the accuracy of target recognition is greatly reduced, resulting in a decrease in the accuracy of distance measurement, which affects the safe driving of the vehicle. Distance measurement by binocular cameras is mainly realized based on the principle of triangulation, and does not need to recognize the target object, but has the disadvantages of complex structure, high algorithm complexity, high cost, especially huge calculation amount, etc., and weather, light irradiation, etc. also have a certain effect on the binocular camera, which affects the accuracy of distance measurement and affects the safe driving of the vehicle.

[0016] In order to solve the problem that the detection error of the position and posture of a traffic sign is large and the accuracy of distance measurement is low in the related art, which affects the safe driving of a vehicle, the embodiment of the present disclosure provides a traffic sign detection method, which constructs an optimization model based on an image frame sequence collected by an on-board camera and odometer frame information corresponding to each frame image in the image frame sequence, and realizes prediction of the position and posture of a traffic sign or the distance between the on-board camera and a traffic sign based on an optimal solution of the optimization model. By optimizing the detection of the position and posture of a traffic sign or the distance between the on-board camera and a traffic sign based on multi-frame images and corresponding odometer frame information, the detection error of the position and posture of a traffic sign or the distance between the on-board camera and a traffic sign can be reduced, and decisions can be made based on more accurate position and posture or distance, improving the accuracy of the decision-making result, and ensuring the safe driving of a vehicle. Exemplary System

[0017] A traffic sign is a sign installed on a road to instruct traffic participants such as vehicles and pedestrians. In the field of intelligent driving, an intelligent driving vehicle controls the vehicle to run normally on the road based on the information shown on the traffic sign. The position, orientation, and distance between the vehicle and the traffic sign affect the recognition accuracy of the sensing module in the intelligent driving system, and further affect the decision-making result of the decision-making module. Therefore, accurately detecting the position, attitude, and distance of the traffic sign is a necessary condition for the safe running of the vehicle. For example, when a vehicle is going to run through an intersection section, the distance from the vehicle to the intersection is predicted, and whether or not the vehicle speed needs to be adjusted is determined based on the predicted distance. The traffic sign in the embodiment of the present disclosure is an example of a traffic signal, and the distance between the vehicle-mounted camera and the traffic signal is predicted. Here, the distance between the vehicle-mounted camera and the traffic signal means the distance between the optical center of the vehicle-mounted camera and the center point of the traffic signal.

[0018] FIG. 1A is a schematic diagram of a configuration of a traffic sign detection system according to one exemplary embodiment of the present disclosure. As shown in FIG. 1A, the detection system includes an in-vehicle camera 101, a drive recorder 102, and a detection device 103.

[0019] The vehicle-mounted camera 101 is used to collect multi-frame images of traffic signs, and can be installed at the center position of the upper end of the windshield inside the vehicle. At present, in the field of intelligent driving, the vehicle-mounted camera applied is mainly a monocular camera or a binocular camera. The monocular camera has advantages such as simple structure, mature algorithm, small calculation amount, and low cost, but is sensitive to weather and light irradiation, etc., and the accuracy of target recognition is greatly reduced in an environment with poor weather and light irradiation. The binocular camera mainly realizes target detection based on the principle of triangulation, and can improve the recognition accuracy to a certain extent compared with the monocular camera, but has disadvantages such as complex structure, high algorithm complexity, and high cost, and also has a huge amount of calculation compared with the monocular camera, which affects the recognition speed. In the embodiment of the present disclosure, the vehicle-mounted camera 101 may be a monocular camera that captures a planar image using a single photosensitive element.

[0020] The driving recorder 102 is commonly known as the black box of a car and is generally installed at the lower end of the passenger seat. It is an electronic recording device that records and stores the vehicle's driving speed, time, mileage and other status information related to the driving of the vehicle, and realizes data output through an interface.

[0021] The detection device 103 is used to determine odometer frame information corresponding to each frame image in the image frame sequence based on the image frame sequence of the traffic sign collected by the on-board camera 101 and the odometer frame information recorded by the drive recorder 102, then determine a residual function and an optimization model based on each frame image and the odometer frame information corresponding to each frame image, further determine a target position of the traffic sign in a predetermined coordinate system based on the residual function and the optimization model, and finally determine a target position and attitude of the traffic sign or a distance between the on-board camera and the traffic sign based on the target position.

[0022] A traffic sign detection system according to an embodiment of the present disclosure detects the position and attitude of a traffic sign or the distance between the on-board camera and a traffic sign based on multi-frame images of a traffic sign collected by an on-board camera and odometer frame information corresponding to the multi-frame images, and can reduce detection errors of the position and attitude of a traffic sign or the distance between the on-board camera and a traffic sign, make decisions based on a more accurate position and attitude or distance, improve the accuracy of decision-making results, and ensure safe driving of the vehicle.

[0023] FIG 1B is a schematic diagram of a distance measurement scenario by a distance measurement system according to one exemplary embodiment of the present disclosure. In the driving scenario shown in FIG 1B, the vehicle-mounted camera 101 performs real-time image collection of a traffic sign (i.e., a traffic light in FIG 1B) during driving, where one frame of the collected traffic light image is as shown in FIG 1C. The vehicle-mounted camera 101 transmits the collected multi-frame image to the detection device 103. The detection device 103 receives the multi-frame image transmitted from the vehicle-mounted camera, and sorts the multi-frame images in order of collection time to form an image frame sequence.

[0024] When the vehicle-mounted camera 101 collects each frame image, the position and posture of the vehicle are different, and the detection device 103 determines odometer frame information corresponding to each frame image of the image frame sequence from the odometer information recorded in the drive recorder 102 based on the corresponding relationship. Then, the detection device 103 constructs a residual function and an optimization model based on each frame image and the odometer frame information corresponding to each frame image, determines an optimal solution of the optimization model, and obtains the world coordinates of the traffic light. Since the relative position between the vehicle-mounted camera 101 and the vehicle is fixed, the real-time coordinates of the vehicle-mounted camera 101 can be determined based on the odometer frame information of the vehicle and the relative position between the vehicle-mounted camera 101 and the vehicle, and finally, the real-time distance between the vehicle-mounted camera 101 and the traffic light is determined based on the real-time coordinates of the vehicle-mounted camera 101 and the world coordinates of the traffic light. Exemplary Methods

[0025] 2 is a flowchart of a traffic sign detection method according to an exemplary embodiment of the present disclosure. As shown in FIG. 2, the detection method includes steps S201 to S204.

[0026] In step S201, an image frame sequence of a traffic sign collected by an on-board camera of a vehicle and odometer frame information corresponding to each frame image in the image frame sequence are determined.

[0027] Exemplarily, the detection method of the embodiment of the present disclosure may be performed by a distance measuring device. When the detection method is used to detect the position and orientation of a traffic sign, the detection device may be a device capable of detecting the position and orientation of a traffic sign. In the following embodiment, when detecting the position and orientation of a traffic sign, the traffic sign is an example of a traffic sign board.

[0028] When the detection method is used to detect the distance between the vehicle-mounted camera and a traffic sign, the detection device may be a distance measuring device that can obtain an image frame sequence collected by the vehicle-mounted camera in real time and odometer frame information corresponding to each frame image in the image frame sequence. In the following embodiment, the vehicle-mounted camera is used as an example to detect the distance between the vehicle-mounted camera and a traffic sign, and the traffic sign is a traffic light.

[0029] An on-board camera installed in a vehicle collects images of a traffic sign to obtain a multi-frame image. The collection times of the multi-frame images have a time sequence relationship, and can be sorted in order of collection time to form an image frame sequence. If the traffic sign is a traffic sign board, one frame image of the traffic sign board collected by the on-board camera is as shown in FIG. 3A. In the embodiment of the present disclosure, the on-board camera may be a monocular camera.

[0030] When the on-board camera collects each frame image in the image frame sequence, the position and attitude (including position and attitude) of the vehicle are different, that is, when the on-board camera collects each frame image, it corresponds to one position and attitude of the vehicle, and the corresponding one frame of odometer information is recorded in the drive recorder of the vehicle at the position and attitude. In the embodiment of the present disclosure, each frame information corresponding to each frame image in the multi-frame information of the odometer information is the odometer frame information corresponding to each frame image.

[0031] In step S202, a residual function and an optimization model are determined based on each frame image and the odometer frame information corresponding to each frame image.

[0032] The residual (denoted as e) represents the difference between the actual observed value and the calculated predicted value. Here, the actual observed value is the actual measurement value or observed value of the image of the traffic sign formed in each frame image, and the predicted value is an estimated value calculated by coordinate transformation or the like from the coordinates of the traffic sign in a preset coordinate system. Here, the preset coordinate system may be a world coordinate system or may be a preset coordinate system.

[0033] The residual e may be a residual of different indices, where different indices correspond to different constraint conditions, and based on this, multiple residual functions (J e For example, we construct the residual function J e may be constructed based on a general optimization function (such as the least squares method).

[0034] The optimization model (denoted as J) is a residual function J e This is an optimization function built on the

[0035] In step S203, a target position of the traffic sign in a preset coordinate system is determined based on the residual function and the optimization model.

[0036] Illustratively, the preset coordinate system is taken as an example to be a world coordinate system. When detecting the position and orientation of a traffic sign, multiple points on the traffic sign may be selected as detection points, and the target position of the traffic sign in the preset coordinate system may be the corresponding target coordinates of the multiple detection points of the traffic sign in the world coordinate system. When determining the target coordinates of each detection point, the optimization goal is determined to be the minimum residual based on the residual function and the optimization model, and then the optimization model is solved (for example, according to an optimization method such as the gradient descent method, Newton's method, conjugate gradient method, and genetic algorithm) with the minimum residual as the optimization goal, and the obtained optimal solution is the target coordinate of the i-th detection point on the traffic sign. In an embodiment of the present disclosure, the i-th detection point X of the traffic sign in the world coordinate system W is determined as the target coordinate of the i-th detection point on the traffic sign. wi The target coordinate of (x wi ,y wi ,zwi ), where i=1, 2, ..., m, m represents the number of detection points on the traffic sign board, and m is a positive integer.

[0037] For example, when detecting the distance between the vehicle-mounted camera and the traffic sign, the target position of the traffic signal in the world coordinate system may be determined based on the residual function and the optimization model. For example, the world coordinate X of the center point of the traffic signal in the world coordinate system W may be expressed as w (x w ,y w ,z w ), where the coordinate is the target position to be determined. When determining the target position, the residual function and the optimization model are used to determine the optimization goal of minimizing the residual, and then the optimization model is solved to obtain the world coordinate X w Obtain the target position of the traffic signal, which is the optimal solution of

[0038] In step S204, a target position and orientation of the traffic sign or a real-time distance between the vehicle-mounted camera and the traffic sign is determined based on the target position.

[0039] For example, when detecting the position and orientation of a traffic sign, information such as the plane on which the traffic sign is located, the orientation and position of the traffic sign, etc. are determined based on the target coordinates corresponding to each of the multiple detection points of the traffic sign determined in step S203, to obtain the target position and orientation of the traffic sign.

[0040] For example, when detecting the distance between the on-board camera and a traffic sign, the real-time distance between the on-board camera and the traffic sign may be determined based on odometer frame information corresponding to each frame image and a target position of the traffic sign in the world coordinate system.

[0041] In some examples, the dashcam is fixedly installed on the vehicle, so that the position of the dashcam relative to the vehicle is fixed. The real-time position of the vehicle in the world coordinate system W can be determined by the vehicle's dashcam, so that the real-time position of the vehicle and the position of the dashcam installed on the vehicle can be combined to determine the real-time position of the dashcam in the world coordinate system W. After obtaining the real-time position of the dashcam and the target position of the traffic light in the world coordinate system W, the real-time distance between the dashcam and the traffic light can be calculated based on a point-to-point distance formula.

[0042] In some examples, the real-time distance between the dashcam and the traffic light is the linear distance d, the lateral distance d x , vertical distance d z , and the vertical distance d y The linear distance d means the distance between the optical center of the vehicle-mounted camera and the center point of the traffic light, and the lateral distance d x means the horizontal distance between the optical center of the camera and the center point of the traffic light in the parallel direction, and the vertical distance d z means the depth distance between the optical center of the camera and the center point of the traffic light in the direction of travel, and the vertical distance d y means the vertical distance between the optical center of the camera and the center point of the traffic light in the height direction, where d is the linear distance and d is the horizontal distance x , vertical distance d z , and the vertical distance d y satisfies the relationship shown in the following formula (1).

number

[0043] A traffic sign detection method according to an embodiment of the present disclosure optimizes targets based on multi-frame images and odometer frame information collected in real time, and determines the position and attitude of a traffic sign or the real-time distance between an on-board camera and a traffic sign, thereby reducing detection errors of the position and attitude or real-time distance of a traffic sign, improving detection accuracy, making decisions based on more accurate position and attitude or real-time distance, improving the accuracy of the decision results, and ensuring safe driving of the vehicle.

[0044] As shown in FIG. 4, based on the embodiment shown in FIG. 2, the above step 201 may include the following steps S2011 to S2013.

[0045] In step S2011, a sequence of image frames of traffic signs collected by an on-board camera of a vehicle is determined.

[0046] Each frame image collected by the vehicle-mounted camera corresponds to one first time stamp. In one example, the vehicle-mounted camera may directly collect a video, and the detection device or distance measurement device selects all or some of the frame images from the multi-frame images included in the video to form an image frame sequence. In another example, the vehicle-mounted camera may collect the multi-frame images at a preset collection period. In yet another example, the vehicle-mounted camera may collect the multi-frame images at random time intervals.

[0047] In the embodiment of the present disclosure, the image frame sequence may be collected in real time by the vehicle-mounted camera. In other application scenarios, the image frame sequence may be collected in non-real time, for example, when determining whether a traffic sign board on a road has a situation such as tilt or distortion, there is no need to determine the position and orientation of the traffic sign board in real time, and in this scenario, the image frame sequence may be images collected by the vehicle-mounted camera at a certain time in the past.

[0048] In step S2012, odometer information collected in real time by the vehicle's dashcam is determined.

[0049] A drive recorder installed in a vehicle is commonly called a black box of a vehicle. It is an electronic recording device that records and stores the vehicle's driving speed, time, mileage, and other status information related to the driving of the vehicle, and realizes data output through an interface. The drive recorder collects and stores odometer information in real time while driving. Each frame of information included in the odometer information collected by the drive recorder is continuous in time sequence, and the shorter the time interval for continuous collection, the more frame information is collected per unit time. For example, if the drive recorder records one frame of information every 20 μs (microseconds), the odometer information for a time length of 1 s (second) includes 50,000 frame information, and if the drive recorder records one frame of information every 1 ms (milliseconds), the odometer information for a time length of 1 s includes 1,000 frame information.

[0050] When the vehicle position and orientation needs to be detected or the distance needs to be measured, the detection device reads odometer information collected in real time from a drive recorder. In an embodiment of the present disclosure, each frame information in the odometer information includes the position and orientation of the vehicle when the drive recorder acquires the corresponding frame information, and the position and orientation of the vehicle includes at least the position of the vehicle, which is the coordinate position of the vehicle in a world coordinate system W. Each frame information in the odometer information corresponds to one second timestamp.

[0051] In step S2013, odometer frame information corresponding to each frame image is determined from the odometer information based on the first timestamp of each frame image in the image frame sequence and the second timestamp of each frame information in the odometer information.

[0052] In order to compare the actual observed value and the predicted value of the same parameter in the same frame image when determining the residual error e, it is necessary to align each frame image in the image frame sequence with the corresponding odometer frame information in time, that is, determine the odometer frame information corresponding to each frame image in the image frame sequence from multiple frame information of the odometer frame information, and then determine the actual observed value and the predicted value based on the frame image and the odometer frame information collected at the same time point.

[0053] When performing temporal alignment, based on the first timestamp of each frame image and the second timestamp of each frame information in the odometer information, the odometer frame information with the same timestamp as each frame image can be determined as the odometer frame information corresponding to each frame image.

[0054] In practical applications, in addition to the method of realizing temporal alignment based on timestamps, alignment between each frame image in an image frame sequence and each frame information in odometer information may also be realized based on other methods such as a synchronous clock, and is not limited to the embodiments of the present disclosure.

[0055] In an embodiment of the present disclosure, by performing data alignment between each frame image in the image frame sequence and each frame information in the odometer information, it is possible to ensure consistency between each frame image and its corresponding odometer frame information, thereby improving the accuracy of the detection results.

[0056] 5 and 6, a method for detecting the position and orientation of a traffic sign will be described by way of example in which the detection method according to the embodiment of the present disclosure is used to detect the position and orientation of a traffic sign. Note that when detecting the position and orientation of a traffic sign, the target positions determined in step S203 above are target coordinates in a preset coordinate system that correspond to each of a plurality of detection points on the traffic sign.

[0057] As shown in FIG. 5, based on the above embodiment, the above step S202 may include the following steps S2021 to S2023.

[0058] In step S2021, feature extraction is performed for each frame image, and detection points of traffic signs in each frame image are determined.

[0059] A feature extraction process is performed on each frame image using a pre-trained image processing model to obtain traffic sign detection points in each frame image. The number of traffic sign detection points is at least one.

[0060] Illustratively, the image processing model may be a model capable of recognizing corner points of a traffic sign, and the corner points of the traffic sign are the detection points. In the embodiment of the present disclosure, the corner points may refer to the intersections of lines in different directions, and are also referred to as vertices. For example, the corner points of a rectangular traffic sign board are the four vertices of the rectangle, and the corner points of a triangular traffic sign board are the three vertices of the triangle. For example, feature extraction is performed on the image frame shown in FIG. 3A to determine a plurality of detection points (for example, detection points 301, 302, 303, and 304 shown in FIG. 3B) on the traffic sign in the frame image.

[0061] For example, the image processing model may be a model capable of recognizing feature points of a traffic sign, and the feature points of the traffic sign are set as detection points, for example, the center point of a traffic sign board, or the midpoint of a side of the traffic sign board.

[0062] In step S2022, a residual function is determined based on the detection points and odometer frame information corresponding to each frame image.

[0063] First, a detection point in each frame image is set as an observation target, and pixel coordinate observation values ​​of the detection point in the pixel coordinate system corresponding to each frame image are determined. The pixel coordinate observation values ​​are actual measurements and are also called actual observation values. The world coordinates X wi is coordinate-transformed to determine predicted pixel coordinate values ​​of the detection points on the traffic sign in the pixel coordinate system corresponding to each frame image. The predicted pixel coordinate values ​​are estimated predicted values. Then, a residual e of the detection point is determined based on the actual observed value and predicted value of the same detection point in the same frame image. Finally, a residual function J is calculated based on the residuals e corresponding to m detection points in n frame images. e where m and n are both positive integers, m represents the number of detection points in one frame image, and n represents the number of image frames included in the image frame sequence.

[0064] In step S2023, an optimization model is determined based on the residual function.

[0065] The optimization model J is the residual function J e The optimization function is constructed based on the residual function J e In the case of multiple, the optimization model J is a set of multiple residual functions J as shown in the following formula (2). e It may be a weighted sum of

number

[0066] In practical applications, the optimization model constructed based on the residual function may be in other forms, such as weighted sum of squares, and is not limited here.

[0067] An embodiment of the present disclosure performs feature extraction on each frame image to determine multiple detection points on traffic signs in each frame image. Since the data amount of the multiple detection points is much smaller than the data amount of the entire frame image, performing residual calculation based on the multiple detection points can greatly reduce the amount of calculation, thereby shortening the recognition time and improving the vehicle decision-making efficiency.

[0068] In practical application, the residual function may include at least one residual function constructed with different constraints. Illustratively, the residual function includes a reprojection residual function and / or a consistency residual function. As shown in FIG. 6, based on the above embodiment, step S2022 may include the following steps S20221 to S20223.

[0069] In step S20221, a reprojection residual function is determined based on the detection points and odometer frame information corresponding to each frame image.

[0070] Theoretically, the observed position of a point on a traffic sign in each frame image should be consistent with the projected position of that point on the same frame image. Based on the constraint that the projected positions must be consistent, we construct a reprojection residual function, J e1 It is written as follows.

[0071] In one implementation, the reprojection residual function may be determined by the following steps.

[0072] First, the pixel coordinates of each detection point in each frame image are determined. One frame image is selected from the multi-frame image as a reference frame image, and images in the multi-frame image other than the reference frame image are set as observation frame images. For ease of distinction, the pixel coordinates of each detection point in the reference frame image may be called the reference pixel coordinates, and the pixel coordinates of each detection point in each observation frame image may be called the observation pixel coordinates.

[0073] Next, a projection relationship between the world coordinate system and a pixel coordinate system corresponding to each frame image is determined based on the relationship between the world coordinate system, the camera coordinate system, the image plane coordinate system, and the pixel coordinate system. Based on the projection relationship, the reference pixel coordinates of each detection point are projected onto the world coordinate system to determine the projected world coordinates of each detection point. Furthermore, based on the odometer frame information, the projected world coordinates of each detection point are reprojected onto the pixel coordinate system corresponding to each observation frame image to obtain the reprojected pixel coordinates of each detection point in each observation frame image.

[0074] Finally, the difference between the observed pixel coordinates and the reprojected pixel coordinates of the same detection point in the same frame image is calculated, and the difference is taken as the reprojection residual. Based on the reprojection residual, a reprojection residual function J e1 Build.

[0075] In step S20222, in response to the number of detected points being greater than a preset threshold, a consistency residual function is determined based on the multiple detected points corresponding to each frame image.

[0076] When the number of detection points on a traffic sign board is greater than one, there is a position consistency constraint between the detection points, and a consistency residual function is constructed based on the constraint.

[0077] In one embodiment, since the multiple detection points are located on the same traffic sign, a unique plane can be determined from any three detection points that are not on the same line among the multiple detection points. For example, a unique plane equation can be constructed based on the first coordinate of detection point 301, the second coordinate of detection point 302, and the third coordinate of detection point 303 in FIG. 3B. Any point among the multiple detection points other than the above three detection points (e.g., detection point 304 in FIG. 3B) should theoretically be located on the plane determined based on the three detection points, i.e., the distance between the fourth coordinate of detection point 304 and the determined plane should be 0. Based on the constraint that the multiple detection points are on the same plane and have the same positions, a consistency residual function is constructed, and J e2 The first coordinate, the second coordinate, the third coordinate, and the fourth coordinate are coordinates in the same coordinate system of the detection point 301, the detection point 302, the detection point 303, and the detection point 304, respectively. Exemplarily, the coordinate system may be a world coordinate system, a camera coordinate system, a vehicle body coordinate system, or the like, and is not limited to the embodiment of the present disclosure.

[0078] In another realization, a consistency residual function is constructed based on the constraint that the distances between the multiple detection points of the same traffic sign and the vehicle are equal in theory, and therefore the distances between the multiple detection points of the same traffic sign and the vehicle are equal, and J e3 It may also be written as:

[0079] In step S20223, a residual function is determined based on the reprojection residual function and / or the consistency residual function.

[0080] Illustratively, steps S20221 and S20222 may be selectively performed based on the actual configuration of the residual function. For example, if the residual function includes only a reprojection residual function, only step S20221 may be performed, if the residual function includes only a consistency residual function, only step S20222 may be performed, and if the residual function includes both a reprojection residual function and a consistency residual function, steps S20221 and S20222 are performed.

[0081] In addition to the residual function constructed with the above two constraints, the residual function may include an imaging residual function constructed based on an imaging constraint, or may include other residual functions constructed with other constraints, and is not limited to the embodiments of the present disclosure.

[0082] In an embodiment of the present disclosure, at least one constraint condition of a traffic sign in a multi-frame image can be determined based on the coordinate system projection relationship, the pinhole phenomenon, and the spatial positional relationship of multiple detection points of the same traffic sign, and a residual function under at least one constraint condition can be constructed. Taking into account the residual function of the multidimensional index, the error of the detection result can be further reduced.

[0083] In some embodiments, the above step S20221 may be realized by the following steps S2211 to S2214.

[0084] In step S2211, the observation pixel coordinates of the detection points in each frame image are determined based on the detection points corresponding to each frame image.

[0085] In the embodiment of the present disclosure, the observed pixel coordinates of each detection point in each frame image are expressed as (v ji ,ν ji), where i = 1, 2, ..., m, j = 1, 2, ..., n, m represents the number of detection points on the traffic sign board, i.e., the number of detection points in one frame image, and since the number of detection points determined in step S2021 is at least 1, m ≥ 1, and n represents the number of image frames included in the image frame sequence, and n ≥ 2.

[0086] In practical implementation, the reprojection residual function may be constructed based on the positional coincidence constraint of one detection point, or may be constructed based on the positional coincidence constraint of multiple detection points. When constructing the reprojection residual function based on the positional coincidence constraint of one detection point, the pixel coordinates of the detection point in each frame image are determined, where the observed pixel coordinates of the detection point in each frame image are expressed as follows: (υ j ,ν j ), where j=1,2,...,n.

[0087] In step S2212, a reference frame image is determined from each frame image, and at least one frame image other than the reference frame image is determined as an observation frame image.

[0088] Here, the reference frame image may be any of the frame images in the multi-frame image, and the observation frame image may be some or all of the images in the image frame sequence other than the reference frame image. For example, the first frame image is the reference frame image, and the remaining images in the image frame sequence other than the reference frame image are the observation frame images.

[0089] The observed pixel coordinates of each detection point in the reference frame image are (υ 0i ,ν 0i ), and the observed pixel coordinates of each detection point in each observation frame image are denoted as (υ ji ,ν ji ), where j = 1, 2, …, n-1.

[0090] In step S2213, reprojected pixel coordinates of the detection point in each observation frame image are determined based on the camera parameters of the vehicle-mounted camera, the odometer frame information corresponding to each frame image, and the observation pixel coordinates of the detection point in the reference frame image.

[0091] In an embodiment of the present disclosure, the camera parameters include camera intrinsic parameters, denoted as K. Odometer frame information corresponding to each frame image is used to determine camera extrinsic parameters corresponding to when the camera collects each frame image, the camera extrinsic parameters including a rotation matrix and a translation vector, denoted as T.

[0092] In one implementation form, the reprojected pixel coordinates of the detection point in each observation frame image may be determined by: determining corresponding reference coordinates of the detection point in a predetermined coordinate system based on camera parameters of the on-board camera and the observation pixel coordinates of the detection point in the reference frame image; and determining the reprojected pixel coordinates of the detection point in each observation frame image based on the camera parameters, the reference coordinates corresponding to the detection point, and odometer frame information corresponding to each frame image.

[0093] In the embodiment of the present disclosure, the preset coordinate system may be a world coordinate system or any customized coordinate system. If the preset coordinate system is not a world coordinate system, the preset coordinate system may be transformed into the world coordinate system by rotation and translation.

[0094] The camera internal parameters K are the shipping parameters of the camera, and the camera internal parameters corresponding to each frame image are fixed. The camera external parameters T relate to the camera position and orientation, and the camera external parameters corresponding to each frame image may be determined by the odometer frame information and the position of the on-board camera relative to the vehicle.

[0095] TIFF2025036354000004.tif53156

[0096] Based on the projection relationship between the pixel coordinate system and the world coordinate system shown in the above formula (3), the observed pixel coordinates (υ 0i ,ν 0i ) in the world coordinate system, each projection coordinate is the corresponding reference coordinate of each detection point in the preset coordinate system, and X 0i It is written as follows.

[0097] TIFF2025036354000005.tif61156

[0098] The reference coordinates X of each detection point on the traffic sign 0i After the above is determined, the reference coordinates X of each detection point are calculated based on the odometer frame information that can indicate vehicle driving information and the projection relationship between the pixel coordinate system corresponding to each observation frame image and the world coordinate system shown in the above formula (4). 0i is reprojected onto the pixel coordinate system of each observation frame image, the reprojected pixel coordinates of each detection point in each observation frame image are obtained, and (υ jip ,ν jip ), where j = 1, 2, …, n-1.

[0099] In step S2214, a reprojection residual function is determined based on the observed pixel coordinates and the reprojected pixel coordinates of the detection point in each observed frame image.

[0100] The observed pixel coordinates of each detection point in each observed frame image determined in step S2211 are (υ ji ,ν ji ), and the reprojected pixel coordinates of each detection point in each observation frame image determined in step S2213 are (υ jip ,ν jip ) and the observed pixel coordinates of each detection point (υ ji ,ν ji ) and the reprojected pixel coordinates (υ jip ,ν jip ) and take the reprojection residual (e jiυ ,e jiν) is determined. The reprojection residual (e jiυ ,e jiν ) may be expressed by the following formula (5).

number

[0101] Illustratively, the optimization function in the embodiment of the present disclosure is exemplified as least squares, and the constructed reprojection residual function is as shown in the following equation (6).

number

[0102] When constructing a reprojection residual function based on the position matching constraint of one detection point, the above equation (6) can be simplified to the following equation (7).

number

[0103] In practical applications, other optimization functions may be used to construct the projection residual function, which is not limited here.

[0104] In an embodiment of the present disclosure, the reprojection residual in the multi-frame image of the detection point on the traffic sign is determined based on the projection relationship between the world coordinate system and the pixel coordinate system, and a reprojection residual function is constructed based on the reprojection residual function, thereby reducing the detection error of the position and orientation of the traffic sign.

[0105] Based on the above embodiment, taking four detection points of a traffic sign as an example, the above step S20222 "in response to the number of detection points being greater than a predetermined threshold, determining a consistency residual function based on a plurality of detection points corresponding to each frame image" may be realized by the following steps S2221 to S2224.

[0106] In step S2221, in response to the number of detection points being greater than a preset threshold, a first detection point, a second detection point, a third detection point and a fourth detection point are determined from the multiple detection points corresponding to each frame image.

[0107] Here, the first detection point, the second detection point, and the third detection point corresponding to one frame image are not on the same line.

[0108] When the number of detection points on a traffic sign board is greater than a preset threshold, the multiple detection points are located on the same traffic sign, so that only one plane can be determined from any three detection points that are not on the same straight line among the multiple detection points, and the remaining detection points other than the three detection points that determine the plane should also be on the plane, that is, the distance between the remaining detection points and the plane should be 0. Based on this, it is determined that there is a position matching constraint between the multiple detection points, and the embodiment of the present disclosure constructs a matching residual function based on the constraint.

[0109] Two detection points are randomly selected as the first detection point and the second detection point from among the multiple detection points corresponding to the j-th frame image. A unique straight line can be determined from the two detection points, the first detection point and the second detection point. Then, from among the multiple detection points other than the first detection point and the second detection point, two points that are not located on the straight line are randomly selected as the third detection point and the fourth detection point. The first detection point, the second detection point, and the third detection point are not on the same straight line, and a unique plane can be determined. Here, j=1, 2, ..., n, n represents the number of image frames included in the image frame sequence, and n ≥ 2.

[0110] In addition, the first detection point, the second detection point, the third detection point, and the fourth detection point corresponding to each frame image may be the same four detection points or may be different detection points. For example, the first detection point corresponding to the j-th frame image may be a corner point in the upper left corner of a traffic sign board, and the first detection point corresponding to the j+1-th frame image may be a corner point in the upper left corner of a traffic sign board, or may be another detection point of the traffic sign board. This is not limited in the embodiments of the present disclosure.

[0111] In step S2222, the first pixel coordinate of the first detection point, the second pixel coordinate of the second detection point, the third pixel coordinate of the third detection point, and the fourth pixel coordinate of the fourth detection point in each frame image are determined.

[0112] In the embodiment of the present disclosure, the first pixel coordinates of the first detection point in the j-th frame image are defined as (υ j1 ,ν j1 ), and the second pixel coordinate of the second detection point is denoted as (υ j2 ,ν j2 ), and the third pixel coordinate of the third detection point is denoted as (υ j3 ,ν j3 ), and the fourth pixel coordinate of the fourth detection point is denoted as (υ j4 ,ν j4 ), where j=1, 2, ..., n, n represents the number of image frames included in the image frame sequence, and n >= 2.

[0113] In step S2223, based on the first pixel coordinate of the first detection point, the second pixel coordinate of the second detection point, the third pixel coordinate of the third detection point, the fourth pixel coordinate of the fourth detection point in each frame image and the camera parameters of the in-vehicle camera, the first camera coordinate of the first detection point, the second camera coordinate of the second detection point, the third camera coordinate of the third detection point, and the fourth camera coordinate of the fourth detection point in each frame image are determined.

[0114] In the embodiment of the present disclosure, the camera parameters include camera internal parameters K, which are factory parameters of the camera, and the camera internal parameters corresponding to each frame image are fixed.

[0115] When determining the camera coordinates of each detection point, first, pixel coordinates in the pixel coordinate system are converted into the image plane coordinate system based on the conversion relationship between the pixel coordinate system and the image plane coordinate system to obtain the image coordinates of each detection point. Then, based on the pinhole phenomenon, the image coordinates in the image plane coordinate system of each detection point in each frame image are converted into the camera coordinate system to obtain the camera coordinates of each detection point. The camera coordinates (x ji ,y ji ,z ji ) may be expressed by the following formula (8).

number

[0116] In step S2224, a consistency residual function is determined based on the first camera coordinate of the first detection point, the second camera coordinate of the second detection point, the third camera coordinate of the third detection point, and the fourth camera coordinate of the fourth detection point in each frame image.

[0117] In determining the consistency residual function, one implementation is as follows: first, determine an image plane equation of the traffic sign board corresponding to each frame image based on the three camera coordinates of the three non-collinear detection points in each frame image; then, determine the distance from the fourth camera coordinate of the fourth detection point to the image plane according to the point-plane distance formula, and the distance is the consistency residual, e j Finally, the consistency residual e j Specifically, this is realized by the following steps S22241 to S22243.

[0118] In step S22241, a traffic sign image plane equation corresponding to each frame image is determined based on the first camera coordinate of the first detection point, the second camera coordinate of the second detection point, and the third camera coordinate of the third detection point in each frame image.

[0119] TIFF2025036354000010.tif103156

[0120] In step S22242, a consistency residual corresponding to each frame image is determined based on the fourth camera coordinate of the fourth detection point in each frame image and the traffic sign image plane equation corresponding to each frame image.

[0121] The fourth camera coordinate of the fourth detection point in the j-th frame image is P4=(x j4 ,y j4 ,z j4 ) and the traffic sign image plane equation corresponding to the j-th frame image is x +B y +C z = 0, and the corresponding match residual e corresponding to the jth frame image is determined according to the point-plane distance formula. j may be expressed by the following formula (10).

number

[0122] In step S22243, a consistency residual function is constructed based on the consistency residual corresponding to each frame image.

[0123] Illustratively, the optimization function in the embodiment of the present disclosure is exemplified as least squares, and the constructed consistency residual function is as shown in the following formula (11).

number

[0124] TIFF2025036354000013.tif65156

[0125] In step S22244, the image plane normal vector of the traffic sign corresponding to each frame image is determined based on the first camera coordinate of the first detection point, the second camera coordinate of the second detection point, and the third camera coordinate of the third detection point in each frame image.

[0126] TIFF2025036354000014.tif93156

[0127] In step S22245, a consistency residual corresponding to each frame image is determined based on the fourth camera coordinate of the fourth detection point in each frame image and the traffic sign image plane normal vector corresponding to each frame image.

[0128] TIFF2025036354000015.tif90156

[0129] In step S22246, a consistency residual function is constructed based on the consistency residual corresponding to each frame image.

[0130] Illustratively, the optimization function in the embodiment of the present disclosure is exemplified as least squares, and the constructed consistency residual function is as shown in the following equation (14).

number

[0131] In practical applications, the consistency residual e j When determining, it may be determined according to the steps shown in steps S22241 and S22242 above, or according to the steps shown in steps S22244 and S22245 above, and when constructing the consistency residual function, it may be constructed based on least squares as shown in step S22243 or step S22246 above, or may be constructed using other optimization functions, and the embodiments of the present disclosure are not limited to any of the above.

[0132] In the embodiment of the present disclosure, the camera coordinates of the multiple detection points of the traffic sign are determined based on the pinhole phenomenon and the transformation relationship between the pixel coordinate system, the image plane coordinate system, and the camera coordinate system, a consistency residual function is constructed based on the constraint that the images of the multiple detection points are on the same plane, and the position and orientation of the traffic sign are determined according to the multiple points in each frame image being on the same plane, so that the detection error of the position and orientation of the traffic sign can be further reduced. The intelligent driving system can make decisions based on the position and orientation of the traffic sign with less error, improve the accuracy of the decision result, and thereby improve the safety of vehicle driving.

[0133] Hereinafter, a method for measuring the distance to a traffic sign will be described with reference to Fig. 7 to Fig. 9, taking as an example a detection method according to an embodiment of the present disclosure that is used to detect a real-time distance between an in-vehicle camera and a traffic sign. Note that when detecting the distance between the in-vehicle camera and a traffic sign, the target position determined in step S203 above is the target position of the traffic sign in the world coordinate system.

[0134] As shown in FIG. 7, based on the above embodiment, step S202 may include the following steps S2024 to S2026.

[0135] In step S2024, feature extraction is performed on each frame image, and the bounding box of the traffic sign in each frame image is determined.

[0136] Each frame image is processed using a pre-trained image processing model to obtain the bounding box of the traffic sign in each frame image. For example, feature extraction is performed on the frame image shown in Figure 1C, and the bounding box of the traffic signal in the frame image is determined as shown by the dashed box 104 in Figure 1C.

[0137] In step S2025, a residual function is determined based on the bounding box and odometer frame information corresponding to each frame image.

[0138] First, in each frame image, the bounding box is taken as the observation object, and the actual measurement value or the actual observation value in the pixel coordinate system of the bounding box is determined. Based on the odometer frame information, the world coordinates X w Then, a coordinate transformation is performed on the traffic light to determine the predicted value of the traffic light in the pixel coordinate system. Then, the residual e is determined based on the actual observed value and the predicted value. Finally, the residual function J is calculated based on the residual e. e Build.

[0139] In step S2026, an optimized model is determined based on the residual function.

[0140] The optimization model J is the residual function J e The optimization function is constructed based on the residual function J e If there are multiple residual functions J, the optimization model J e It may be a weighted sum of

[0141] An embodiment of the present disclosure performs feature extraction on each frame image, and determines the bounding box of the traffic sign in each frame image. Since the data amount of the bounding box is much smaller than the data amount of the entire frame image, performing residual calculation based on the bounding box can reduce the amount of calculation, thereby shortening the time required for distance prediction and further improving the efficiency of vehicle decision-making.

[0142] In practical application, the residual function may include at least one residual function constructed with different constraints. Illustratively, the residual function includes at least one of a projection residual function, an imaging residual function and a consistency residual function. As shown in FIG. 8, based on the above embodiment, step S2025 may include the following steps S20251 to S20254.

[0143] In step S20251, a projection residual function is determined based on the bounding box and odometer frame information corresponding to each frame image.

[0144] Theoretically, the position of a point on a traffic sign in each frame image should match the projected position of that point. Based on the constraint that the image positions match, a projection residual function is constructed, and J e1 It is written as follows.

[0145] In one implementation, first, at least one observation point is preset on the traffic sign. Next, the observation pixel coordinates of each observation point are determined within the bounding box. Based on the odometer frame information, the world coordinates of each observation point in the world coordinate system W are projected onto each frame image to determine the projected pixel coordinates of each observation point. Finally, the difference value between the observation pixel coordinates and the projected image pixel coordinates is set as the projection residual, and a projection residual function J is calculated based on the projection residual. e1 Build.

[0146] In step S20252, an imaging residual function is determined based on the bounding box and odometer frame information corresponding to each frame image.

[0147] Theoretically, the dimensions of a line in a traffic sign in each frame image should match the projected dimensions of the line. Based on the constraint that the imaging dimensions match, an imaging residual function is constructed, and J e2 It is written as follows.

[0148] In one implementation, first, at least one observation line is set on the traffic sign. Then, the observed pixel size of each observation line is determined within the bounding box. Based on the odometer frame information, the actual size of each observation line in the world coordinate system W is projected onto each frame image to determine the imaged pixel size of each observation line. Finally, the difference value between the observed pixel size and the imaged pixel size is taken as the imaging residual, and an imaging residual function is constructed based on the imaging residual, and J e2 It is written as follows.

[0149] In step S20253, in response to the number of traffic signs being greater than a preset threshold, a consistency residual function is determined based on the bounding box and odometer frame information corresponding to each frame image.

[0150] When the number of traffic signs is greater than one, the distances between the multiple traffic signs and the vehicle are the same. Therefore, theoretically, multiple traffic signs located on the same plane in the same frame image have the same predicted distance from the vehicle-mounted camera, and if the heights of multiple traffic signs are the same, the predicted heights of each traffic sign are also the same. Based on the constraint that the distances and heights of multiple traffic signs are the same, a consistency residual function is constructed, and J e3 It is written as follows.

[0151] In step S20254, a residual function is determined based on at least one of a projection residual function, an imaging residual function, and a consistency residual function.

[0152] Illustratively, steps S20251, S20252, and S20253 may be selectively performed according to the actual composition of the residual function. For example, when the residual function includes only a projection residual function, only step S20251 may be performed; when the residual function includes an imaging residual function and a consistency residual function, only step S20252 and S20253 may be performed; when the residual function includes a projection residual function, an imaging residual function, and a consistency residual function, step S20251, step S20252, and step S20253 may be performed.

[0153] In addition to the three residual functions constructed with the above three constraints, the residual functions may include other residual functions constructed with other constraints, and are not limited to the embodiments of the present disclosure.

[0154] In an embodiment of the present disclosure, at least one constraint condition in a multi-frame image of a traffic sign is determined based on the coordinate system projection relationship, the pinhole phenomenon, and the spatial positional relationship of multiple traffic signs arranged side by side, and a residual function under the at least one constraint condition is constructed. Taking into account the residual function of the multidimensional index, the error of the predicted distance can be further reduced.

[0155] Based on the above embodiment, taking the example of pre-setting one observation point on a traffic sign, the above step S20251, "determining a projection residual function based on the bounding box and odometer frame information corresponding to each frame image", may be realized by the following steps S2511 to S2514.

[0156] In step S2511, the observation pixel coordinates of a preset observation point of a traffic sign in each frame image are determined based on the bounding box corresponding to each frame image.

[0157] The preset observation point may be any point on the traffic sign, for example, the preset observation point may be the center point, the upper left corner point, a specific point, etc. In the following, an example will be described in which the preset observation point is the center point of a traffic signal.

[0158] From the geometrical relationship, it can be seen that the center point of the traffic signal corresponds to the center point of the bounding box. Based on this, the coordinates of the center point of the bounding box are determined as the observed pixel coordinates of the preset observation point, and are denoted as (υ, ν). The observed pixel coordinates of the preset observation point in the j-th frame image are denoted as (υ j ,ν j ), where j=1, 2, ..., n, n is the number of image frames included in the image frame sequence, and n >= 2.

[0159] In step S2512, the world coordinates of a preset observation point are determined.

[0160] The world coordinates of the preset observation point are the coordinates of the traffic light center point (x w ,y w ,z w ), whose coordinates are the target position to be determined.

[0161] In step S2513, the projected pixel coordinates of a preset observation point in each frame image are determined based on the camera parameters of the vehicle-mounted camera, the odometer frame information corresponding to each frame image, and the world coordinates of the preset observation point.

[0162] In an embodiment of the present disclosure, the camera parameters include camera intrinsic parameters, denoted as K. Odometer frame information corresponding to each frame image is used to determine camera extrinsic parameters, which include a rotation matrix and a translation vector, denoted as T.

[0163] In one implementation form, the projected pixel coordinates of a predetermined observation point in the jth frame image may be determined by: determining a projection relationship between a world coordinate system and a pixel coordinate system corresponding to each frame image based on camera parameters of an on-board camera and odometer frame information corresponding to each frame image; and determining the projected pixel coordinates of the predetermined observation point in each frame image based on the projection relationship and the world coordinates of the predetermined observation point.

[0164] TIFF2025036354000017.tif78156

[0165] TIFF2025036354000018.tif37156

[0166] In another implementation form, the projected pixel coordinates of the preset observation point in the jth frame image may be determined by: determining the camera coordinates of the preset observation point in each frame image based on the world coordinates of the preset observation point and odometer frame information corresponding to each frame image; and determining the projected pixel coordinates of the preset observation point in each frame image based on the camera parameters of the on-board camera and the camera coordinates of the preset observation point in each frame image.

[0167] TIFF2025036354000019.tif59156

[0168] Let the camera parameters (internal camera parameters) be K, and the camera coordinates of the preset observation point be (x j ,y j ,z j ), the projected pixel coordinates (υ jp ,ν jp ) may be expressed by the following equation (17).

number

[0169] In step S2514, a projection residual function is determined based on the observation pixel coordinates and the projected pixel coordinates of a preset observation point in each frame image.

[0170] The observation pixel coordinates of the predicted observation point in each frame image determined in step S2511 are (υ j ,ν j ), and the projected pixel coordinates of the predicted observation point in each frame image determined in step S2513 are (υ jp ,ν jp ) and the observed pixel coordinates (υ j ,ν j ) and the projected pixel coordinates (υ jp ,ν jp ) is determined based on the projection residual (e jυ ,νjν ) may be expressed by the following formula (18).

number

[0171] Illustratively, the optimization function in the embodiment of the present disclosure is exemplified as least squares, and the constructed projection residual function is as shown in the following equation (19).

number

[0172] In practical applications, other optimization functions may be used to construct the projection residual function, which is not limited here.

[0173] In an embodiment of the present disclosure, a projection constraint condition in a multi-frame image of a traffic sign is determined based on the projection relationship between the world coordinate system and the pixel coordinate system, and the error in the predicted distance between the on-board camera and the traffic sign can be reduced.

[0174] Based on the above embodiment, taking the example of pre-setting one observation line on a traffic sign, the above step S20252 of "determining an imaging residual function based on the bounding box and odometer frame information corresponding to each frame image" may be realized by the following steps S2521 to S2524.

[0175] In step S2521, the observation pixel size of the preset observation line of the traffic sign in each frame image is determined based on the bounding box corresponding to each frame image.

[0176] The preset observation line may be any line segment of the traffic sign, for example, the preset observation line is the short side of the traffic signal, a diagonal line, or a specific line, etc. In the following, an example will be described in which the preset observation line is the short side of the traffic signal.

[0177] From the geometric relationship, it can be seen that the short side of the traffic signal corresponds to the short side of the bounding box. Based on this, the pixel dimension of the short side of the bounding box is determined as the observation pixel dimension of the preset observation line, which is denoted as s. The observation pixel dimension of the preset observation line in the j-th frame image is denoted as s. j where j=1, 2, . . . , n, n is the number of image frames included in the image frame sequence, and n≧2.

[0178] In step S2522, the actual dimensions of the preset observation line are determined.

[0179] The short side of a traffic signal generally conforms to national standards and is designated by the letter S.

[0180] In step S2523, the imaging pixel size of the preset observation line in each frame image is determined based on the camera parameters of the vehicle-mounted camera, the odometer frame information corresponding to each frame image, and the actual size of the preset observation line.

[0181] In an embodiment of the present disclosure, the camera parameters include camera intrinsic parameters, denoted as K. Odometer frame information corresponding to each frame image is used to determine camera extrinsic parameters, which include a rotation matrix and a translation vector, denoted as T.

[0182] In one implementation, the imaging pixel dimensions of the preset observation line in the jth frame image may be determined by a step of determining a projection relationship between a world coordinate system and a pixel coordinate system corresponding to each frame image based on camera parameters of the on-board camera and odometer frame information corresponding to each frame image, and a step of determining the imaging pixel dimensions of the preset observation line in each frame image based on the projection relationship and the actual dimensions of the preset observation line.

[0183] TIFF2025036354000023.tif38156

[0184] In another implementation form, the imaging pixel dimensions of the preset observation line in the jth frame image may be determined by a step of determining the camera coordinates of a feature point on the preset observation line in each frame image based on the world coordinates of any feature point on the preset observation line and odometer frame information corresponding to each frame image, and a step of determining the imaging pixel dimensions of the preset observation line in each frame image based on the camera parameters of the on-board camera and the camera coordinates of the feature point on the preset observation line in each frame image.

[0185] TIFF2025036354000024.tif60156

[0186] Based on the pinhole phenomenon, the imaging pixel size s in the pixel coordinate system of the preset observation line of the traffic signal jp , the actual dimension S of the preset observation line of the traffic light, the camera parameters (inner parameters of the camera) K of the on-board camera, and the camera coordinate z j satisfies the relationship shown in the following formula (21).

number

[0187] In accordance with equations (20) and (21), the imaging pixel size s of the preset observation line in the jth frame image is jp may be determined and expressed by the following formula (22).

number

[0188] In step S2524, an imaging residual function is determined based on the observation pixel size and imaging pixel size of the preset observation line in each frame image.

[0189] The observation pixel size of the predicted observation line in each frame image determined in step S2521 is sj The imaging pixel size of the predicted observation line in each frame image determined in step S2523 is s jp and the observation pixel size s j and imaging pixel size s jp The imaging residual e of the jth frame image is determined based on js may be expressed by the following equation (23).

number

[0190] Illustratively, the optimization function in the embodiment of the present disclosure is exemplified as the least squares, and the constructed imaging residual function is as shown in the following Equation (24).

number

[0191] In practical applications, other optimization functions may be used to construct the imaging residual function, which is not limited here.

[0192] In an embodiment of the present disclosure, an imaging constraint condition in a multi-frame image of a traffic sign is determined based on the pinhole phenomenon, thereby reducing the error in the predicted distance between the vehicle-mounted camera and the traffic sign.

[0193] Based on the above embodiment, when the number of traffic signs is greater than 1, multiple traffic signs located on the same plane in the same frame image have the same predicted distance from the on-board camera, and if the heights of multiple traffic signs are consistent, the predicted heights of each traffic sign are also the same. Based on this, the above step S20253 of "determining a consistency residual function based on the bounding box and odometer frame information corresponding to each frame image in response to the number of traffic signs being greater than a predetermined threshold" may be realized by the following steps S2531 to S2533.

[0194] In step S2531, in response to the number of traffic signs being greater than a preset threshold, a first traffic sign and a second traffic sign are determined from the plurality of traffic signs.

[0195] In the embodiment of the present disclosure, the preset threshold may be 1, that is, the number of traffic signs is at least two. First, select two from the multiple traffic signs as the first traffic sign and the second traffic sign, for example, select two of the four traffic signals shown in FIG. 1C, and select the first traffic signal from the left as the first traffic signal, and the third traffic signal from the left as the second traffic signal.

[0196] In step S2532, a first camera coordinate of a first traffic sign and a second camera coordinate of a second traffic sign in each frame image are determined based on the bounding box and odometer frame information corresponding to each frame image.

[0197] TIFF2025036354000029.tif96156

[0198] In step S2533, a consistency residual function is determined based on the first camera coordinates of the first traffic sign and the second camera coordinates of the second traffic sign in each frame image.

[0199] TIFF2025036354000030.tif64156

[0200] Illustratively, the optimization function in the embodiment of the present disclosure is exemplified as least squares, and the constructed consistency residual function is as shown in the following equation (28).

number

[0201] TIFF2025036354000032.tif61156

[0202] In practical applications, the optimization model constructed based on the residual function may be in other forms, such as weighted sum of squares, and is not limited here.

[0203] In an embodiment of the present disclosure, when it is determined that multiple traffic signs exist side by side, a consistency constraint condition for the distance and height of the multiple traffic signs is determined based on the spatial positional relationship in which the longitudinal distance and height between the multiple traffic signs and the vehicle are the same, thereby not only reducing the error in the predicted distance between the on-board camera and the traffic signs, but also solving the problem of low consistency in distance measurements of multiple traffic signs, further improving the accuracy of distance measurements, and thereby improving the driving safety of the vehicle.

[0204] As shown in FIG. 9, based on the above embodiment, the above step S204 may be realized by the following steps S2041 and S2042 when determining the real-time distance between the vehicle-mounted camera and the traffic sign.

[0205] In step S2041, the real-time position of the vehicle-mounted camera is determined based on the odometer frame information corresponding to each frame image.

[0206] The real-time position of the vehicle can be determined based on the odometer frame information, and the position of the on-board camera installed in the vehicle is fixed. The real-time position of the on-board camera is determined based on the real-time position of the vehicle and the relative position of the on-board camera to the center point of the vehicle body, and (x jw ,y jw ,z jw ) is written as

[0207] In step S2042, a real-time distance between the dashcam and the traffic sign is determined based on the real-time position of the dashcam and the target position.

[0208] The target position (x w ,y w ,z w) and the real-time position of the onboard camera (x jw ,y jw ,z jw ) based on the formula for the distance between two points, calculate the real-time distance between the dashcam and the traffic light.

[0209] TIFF2025036354000033.tif39156

[0210] In practical applications, the required real-time distance may be determined according to specific application scenarios, and is not limited to the embodiments of the present disclosure.

[0211] In an embodiment of the present disclosure, by predicting the real-time distance between the on-board camera and the traffic sign based on the real-time position of the on-board camera and the target position of the traffic sign, the error of the predicted distance between the on-board camera and the traffic sign can be reduced, the accuracy of distance measurement can be improved, and the safety of vehicle driving can be improved. Exemplary Apparatus

[0212] Based on the above embodiments, an embodiment of the present disclosure provides a traffic sign detection device, in which each module included in the detection device and each unit included in each module may be realized by a processor in a computer device, or may be realized by a specific logic circuit, and in the execution process, the processor may be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0213] FIG. 10 is a schematic diagram of a configuration of a traffic sign detection device according to one exemplary embodiment of the present disclosure. As shown in FIG. 10, the detection device 1000 includes: A first determination module 1001 is used for determining an image frame sequence of a traffic sign collected by an on-board camera of a vehicle, and odometer frame information corresponding to each frame image in the image frame sequence; A second determination module 1002 is used to determine a residual function and an optimization model based on each frame image and the odometer frame information corresponding to each frame image determined by the first determination module 1001; a third determination module 1003, which is used to determine target positions of a plurality of detection points on the traffic sign in a preset coordinate system according to the residual function and the optimization model determined by the second determination module 1002; and a fourth determination module 1004 used to determine a target position attitude of the traffic sign or a real-time distance between the on-board camera and the traffic sign based on the target position determined by the third determination module 1003.

[0214] As shown in FIG. 11, based on the embodiment shown in FIG. 10, the first determining module 1001: a first determination sub-module 10011, which is used to determine a sequence of image frames of traffic signs collected by an on-board camera of a vehicle; a second determining sub-module 10012 used to determine odometer information collected in real time by a dashcam in the vehicle; and a third determination submodule 10013 used to determine odometer frame information corresponding to each frame image from the odometer information based on a first timestamp of each frame image in the image frame sequence and a second timestamp of each frame information in the odometer information.

[0215] 11 , in some embodiments, the second determining module 1002 may include a fourth determining sub-module 10021, a fifth determining sub-module 10022, and a sixth determining sub-module 10023. When the detection device is used to detect the position and orientation of a traffic sign, the functions of the fourth determining sub-module 10021 to the sixth determining sub-module 10023 are different from those when detecting the real-time distance between the in-vehicle camera and the traffic sign.

[0216] When detecting the position and posture of a traffic sign, the fourth determination sub-module 10021 is used to perform feature extraction for each frame image and determine the detection points of the traffic sign in each frame image, the fifth determination sub-module 10022 is used to determine a residual function based on the detection points corresponding to each frame image and odometer frame information, and the sixth determination sub-module 10023 is used to determine an optimization model based on the residual function.

[0217] In some embodiments, when detecting the position and orientation of a traffic sign, the fifth determining sub-module 10022 may include a first determining unit, a second determining unit and a third determining unit.

[0218] The first determination unit is used to determine a reprojection residual function based on the detection points corresponding to each frame image and odometer frame information, the second determination unit is used to determine a consistency residual function based on a plurality of detection points corresponding to each frame image in response to the number of detection points being greater than a predetermined threshold, and the third determination unit is used to determine a residual function based on the reprojection residual function and / or the consistency residual function.

[0219] In some embodiments, the first determining unit comprises: A first determination subunit is used to determine an observation pixel coordinate of a detection point in each frame image based on the detection point corresponding to each frame image; A second determination subunit is used for determining a reference frame image from each frame image, and determining at least one frame image other than the reference frame image as an observation frame image; A third determination subunit is used to determine a reprojected pixel coordinate of the detection point in each observation frame image based on the camera parameters of the vehicle-mounted camera, the odometer frame information corresponding to each frame image, and the observation pixel coordinate of the detection point in the reference frame image; and a fourth determination subunit used to determine a reprojection residual function based on the observed pixel coordinates and the reprojected pixel coordinates of the detection point in each observed frame image.

[0220] In some embodiments, the third determination subunit is further used to determine corresponding reference coordinates of the detection point in a predetermined coordinate system based on camera parameters of the on-board camera and observed pixel coordinates of the detection point in the reference frame image, and to determine reprojected pixel coordinates of the detection point in each observed frame image based on the camera parameters, the reference coordinates corresponding to the detection point, and odometer frame information corresponding to each frame image.

[0221] In some embodiments, the second determination unit comprises: a fifth determination subunit used to determine a first detection point, a second detection point, a third detection point, and a fourth detection point from the plurality of detection points corresponding to each frame image in response to the number of detection points being greater than a preset threshold, wherein the first detection point, the second detection point, and the third detection point corresponding to one frame image are not on the same straight line; a sixth determination subunit used to determine a first pixel coordinate of a first detection point, a second pixel coordinate of a second detection point, a third pixel coordinate of a third detection point, and a fourth pixel coordinate of a fourth detection point in each frame image; a seventh determination subunit used to determine a first camera coordinate of the first detection point, a second camera coordinate of the second detection point, a third camera coordinate of the third detection point, and a fourth camera coordinate of the fourth detection point in each frame image, based on a first pixel coordinate of the first detection point, a second pixel coordinate of the second detection point, a third pixel coordinate of the third detection point, and a fourth pixel coordinate of the fourth detection point in each frame image, and camera parameters of the vehicle-mounted camera; and an eighth determination subunit used to determine a consistency residual function based on the first camera coordinate of the first detection point, the second camera coordinate of the second detection point, the third camera coordinate of the third detection point, and the fourth camera coordinate of the fourth detection point in each frame image.

[0222] In some embodiments, the eighth determination subunit is further used for determining a traffic sign image plane corresponding to each frame image based on the first camera coordinate of the first detection point, the second camera coordinate of the second detection point, and the third camera coordinate of the third detection point in each frame image, determining a consistency residual corresponding to each frame image based on the fourth camera coordinate of the fourth detection point in each frame image and the traffic sign image plane corresponding to each frame image, and determining a consistency residual function based on the consistency residual corresponding to each frame image.

[0223] In some embodiments, when detecting the real-time distance between the on-board camera and the traffic sign, the fourth determination sub-module 10021 is further used to perform feature extraction for each frame image respectively and determine the bounding box of the traffic sign in each frame image, the fifth determination sub-module 10022 is further used to determine a residual function based on the bounding box corresponding to each frame image and the odometer frame information, and the sixth determination sub-module 10023 is further used to determine an optimization model based on the residual function.

[0224] In some embodiments, when detecting the real-time distance between the onboard camera and a traffic sign, the fifth determination sub-module 10022 may include a fourth determination unit, a fifth determination unit, a sixth determination unit and a seventh determination unit.

[0225] The fourth determination unit is used to determine a projection residual function based on the bounding box and odometer frame information corresponding to each frame image, the fifth determination unit is used to determine an imaging residual function based on the bounding box and odometer frame information corresponding to each frame image, the sixth determination unit is used to determine a consistency residual function based on the bounding box and odometer frame information corresponding to each frame image in response to the number of traffic signs being greater than a predetermined threshold, and the seventh determination unit is used to determine a residual function based on at least one of the projection residual function, the imaging residual function, and the consistency residual function.

[0226] In some embodiments, the fourth determination unit is specifically used for determining, based on a bounding box corresponding to each frame image, observation pixel coordinates of a predetermined observation point of a traffic sign in each frame image, determining world coordinates of the predetermined observation point, determining projected pixel coordinates of the predetermined observation point in each frame image based on camera parameters of the on-board camera, odometer frame information corresponding to each frame image and the world coordinates of the predetermined observation point, and determining a projection residual function based on the observation pixel coordinates and projected pixel coordinates of the predetermined observation point in each frame image.

[0227] In some embodiments, the fourth determination unit is specifically used to determine a projection relationship between a world coordinate system and a pixel coordinate system corresponding to each frame image based on camera parameters of the on-board camera and odometer frame information corresponding to each frame image, and to determine projected pixel coordinates of a predetermined observation point in each frame image based on the projection relationship and the world coordinates of the predetermined observation point.

[0228] In some embodiments, the fifth determination unit is specifically used for determining, based on a bounding box corresponding to each frame image, observation pixel dimensions of the preset observation line of the traffic sign in each frame image, determining actual dimensions of the preset observation line, determining imaging pixel dimensions of the preset observation line in each frame image based on camera parameters of the vehicle-mounted camera, odometer frame information corresponding to each frame image and the actual dimensions of the preset observation line, and determining an imaging residual function based on the observation pixel dimensions and imaging pixel dimensions of the preset observation line in each frame image.

[0229] In some embodiments, the fifth determination unit is specifically used for determining a projection relationship between the world coordinate system and a pixel coordinate system corresponding to each frame image based on the camera parameters of the on-board camera and the odometer frame information corresponding to each frame image, and for determining the imaging pixel dimensions of the preset observation line in each frame image based on the projection relationship and the actual dimensions of the preset observation line.

[0230] In some embodiments, the sixth determination unit is specifically used for determining a first traffic sign and a second traffic sign from the multiple traffic signs in response to the number of traffic signs being greater than a preset threshold, determining a first camera coordinate of the first traffic sign and a second camera coordinate of the second traffic sign in each frame image based on the bounding box and odometer frame information corresponding to each frame image, and determining a consistency residual function based on the first camera coordinate of the first traffic sign and the second camera coordinate of the second traffic sign in each frame image.

[0231] In some embodiments, the fourth determination module 1004 is specifically used to determine a real-time position of the dash camera based on odometer frame information corresponding to each frame image, and determine a real-time distance between the dash camera and a traffic sign based on the real-time position of the dash camera and the target position.

[0232] It should be noted that the above description of the exemplary embodiment of the device is similar to the description of the above method, and has the same beneficial effects as the corresponding exemplary embodiment of the method. Those skilled in the art can refer to the description of the exemplary embodiment of the method of the present disclosure to understand the technical details and corresponding beneficial technical effects not disclosed in the exemplary embodiment of the device of the present disclosure, and the description will be omitted here. Exemplary Electronic Devices

[0233] FIG. 12 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. As shown in FIG. 12, an electronic device 1200 includes at least one processor 1201 and a memory 1202.

[0234] The processor 1201 may be a central processing unit (CPU) or other type of processing unit having data processing and / or instruction execution capabilities and may control the other components in the electronic device 1200 to perform desired functions.

[0235] The memory 1202 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache). The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. The computer-readable storage media may store one or more computer program instructions, and the processor 1201 may execute the one or more computer program instructions to realize the traffic sign detection method according to each embodiment of the present disclosure described above and / or other desired functions.

[0236] In one example, the electronic device 1200 may further include input devices 1203 and output devices 1204 connected to each other via a bus system and / or other form of connection (not shown).

[0237] The input device 1203 may include a keyboard, a mouse, and the like.

[0238] The output device 1204 can output various types of information to the outside, and may include, for example, a display, a speaker, a printer, a communication network, and remote output devices connected thereto.

[0239] 12, for the sake of simplicity, only some of the components related to the present disclosure in the electronic device 1200 are shown, and components such as a bus and an input / output interface are omitted. Also, depending on a specific application situation, the electronic device 1200 may further include any other appropriate components. Exemplary Computer Program Products and Computer-Readable Storage Media

[0240] In addition to the above methods and apparatus, embodiments of the present disclosure further provide a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform steps in the traffic sign detection method of each embodiment of the present disclosure described in the "Exemplary Methods" section above.

[0241] The computer program product may be written with program code for carrying out operations of the disclosed embodiments in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and traditional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, as separate software packages, partly on the user's computing device and partly on a remote computing device, or entirely on a remote computing device or server.

[0242] Additionally, an embodiment of the present disclosure may be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform steps in the traffic sign detection method of each embodiment of the present disclosure described in the "Exemplary Method" section above.

[0243] The computer readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0244] Although the basic principles of the present disclosure have been described above with reference to specific embodiments, the advantages, advantages, effects, etc. mentioned in the present disclosure are not limited but merely illustrative, and these advantages, advantages, effects, etc. are not necessarily possessed by each embodiment of the present disclosure. In addition, the specific details disclosed above are not limited but merely serve to serve as examples and to facilitate understanding, and the above details do not necessarily limit the present disclosure to be realized by the above specific details.

[0245] Those skilled in the art can make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and its equivalent technology, the present disclosure is also intended to include these modifications and variations. [CROSS REFERENCE TO RELATED APPLICATIONS]

[0246] This disclosure claims priority to Chinese patent application filed on August 31, 2023, bearing application number 202311117556.4 and entitled "Method, apparatus, storage medium and electronic device for measuring distance to traffic signs," and Chinese patent application filed on August 31, 2023, bearing application number 202311118124.5 and entitled "Method, apparatus, storage medium and electronic device for detecting position and attitude of traffic signs," the entire contents of which are incorporated herein by reference.

Claims

1. determining an image frame sequence of a traffic sign captured by an on-board camera of a vehicle, and odometer frame information corresponding to each frame image in the image frame sequence; determining a residual function and an optimization model based on each of the frame images and odometer frame information corresponding to each of the frame images; determining a target position of the traffic sign in a preset coordinate system based on the residual function and the optimization model; determining a target position and orientation of the traffic sign or a real-time distance between the vehicle-mounted camera and the traffic sign based on the target position.

2. The step of determining odometer frame information corresponding to each frame image in the sequence of image frames comprises: determining odometer information collected in real time by a dashcam of the vehicle; and determining odometer frame information corresponding to each of the frame images from the odometer information based on a first timestamp of each frame image in the image frame sequence and a second timestamp of each frame information in the odometer information.

3. The step of determining a residual function and an optimization model based on each of the frame images and odometer frame information corresponding to each of the frame images includes: performing feature extraction for each of the frame images and determining detection points of the traffic sign in each of the frame images; determining a residual function based on the detection points and the odometer frame information corresponding to each of the frame images; and determining an optimization model based on the residual function.

4. The step of determining a residual function based on the detection points and the odometer frame information corresponding to each of the frame images includes: determining a reprojection residual function based on the detection points and the odometer frame information corresponding to each of the frame images; determining a consistency residual function based on the plurality of detection points corresponding to each of the frame images in response to the number of the detection points being greater than a predetermined threshold; The method of claim 3 , further comprising: determining the residual function based on the reprojection residual function and / or the consistency residual function.

5. The step of determining a reprojection residual function based on the detection points and the odometer frame information corresponding to each of the frame images includes: determining an observation pixel coordinate of the detection point in each of the frame images based on the detection point corresponding to each of the frame images; determining a reference frame image from each of the frame images, and determining at least one frame image other than the reference frame image as an observation frame image; determining reprojected pixel coordinates of the detection point in each observation frame image based on camera parameters of the vehicle-mounted camera, the odometer frame information corresponding to each frame image, and the observation pixel coordinates of the detection point in the reference frame image; The traffic sign detection method according to claim 4 , further comprising: determining a reprojection residual function based on the observed pixel coordinates and the reprojected pixel coordinates of the detection points in each of the observed frame images.

6. determining reprojected pixel coordinates of the detection point in each observation frame image based on camera parameters of the vehicle-mounted camera, the odometer frame information corresponding to each frame image, and the observation pixel coordinates of the detection point in the reference frame image, determining a reference coordinate corresponding to the detection point in the preset coordinate system based on a camera parameter of the vehicle-mounted camera and an observed pixel coordinate of the detection point in the reference frame image; determining reprojected pixel coordinates of the detection point in each observed frame image based on the camera parameters, the reference coordinates corresponding to the detection point, and the odometer frame information corresponding to each of the frame images.

7. The step of determining a consistency residual function based on a plurality of detection points corresponding to each of the frame images in response to the number of the detection points being greater than a predetermined threshold value includes: a step of determining a first detection point, a second detection point, a third detection point and a fourth detection point from the plurality of detection points corresponding to each of the frame images in response to the number of the detection points being greater than a preset threshold value, wherein the first detection point, the second detection point and the third detection point corresponding to one frame image are not on the same straight line; determining a first pixel coordinate of the first detection point, a second pixel coordinate of the second detection point, a third pixel coordinate of the third detection point, and a fourth pixel coordinate of the fourth detection point in each of the frame images; determining a first camera coordinate of the first detection point, a second camera coordinate of the second detection point, a third camera coordinate of the third detection point, and a fourth camera coordinate of the fourth detection point in each of the frame images based on a first pixel coordinate of the first detection point, a second pixel coordinate of the second detection point, a third pixel coordinate of the third detection point, and a fourth pixel coordinate of the fourth detection point in each of the frame images, and camera parameters of the vehicle-mounted camera; and determining a consistency residual function based on a first camera coordinate of the first detection point, a second camera coordinate of the second detection point, a third camera coordinate of the third detection point, and a fourth camera coordinate of the fourth detection point in each of the frame images.

8. determining a consistency residual function based on a first camera coordinate of the first detection point, a second camera coordinate of the second detection point, a third camera coordinate of the third detection point, and a fourth camera coordinate of the fourth detection point in each of the frame images, determining a traffic sign image plane corresponding to each of the frame images based on a first camera coordinate of the first detection point, a second camera coordinate of the second detection point, and a third camera coordinate of the third detection point in each of the frame images; determining a consistency residual corresponding to each of the frame images based on a fourth camera coordinate of the fourth detection point in each of the frame images and a traffic sign image plane corresponding to each of the frame images; The traffic sign detection method according to claim 7, further comprising: determining a consistency residual function based on the consistency residual corresponding to each of the frame images.

9. The step of determining a residual function and an optimization model based on each of the frame images and odometer frame information corresponding to each of the frame images includes: performing feature extraction for each of the frame images and determining a bounding box of the traffic sign in each of the frame images; determining a residual function based on the bounding box and the odometer frame information corresponding to each of the frame images; and determining an optimization model based on the residual function.

10. The step of determining a residual function based on the bounding box and the odometer frame information corresponding to each of the frame images includes: determining a projection residual function based on the bounding box and the odometer frame information corresponding to each of the frame images; determining an imaging residual function based on the bounding box and the odometer frame information corresponding to each of the frame images; determining a consistency residual function based on the bounding box and the odometer frame information corresponding to each of the frame images in response to the number of the traffic signs being greater than a preset threshold; and determining the residual function based on at least one of the projection residual function, the imaging residual function, and the consistency residual function.

11. The step of determining a projection residual function based on the bounding box and the odometer frame information corresponding to each of the frame images includes: determining an observation pixel coordinate of a preset observation point of the traffic sign in each of the frame images based on the bounding box corresponding to each of the frame images; determining world coordinates of the preset observation point; determining projected pixel coordinates of the preset observation point in each of the frame images based on camera parameters of the vehicle-mounted camera, the odometer frame information corresponding to each of the frame images, and world coordinates of the preset observation point; The traffic sign detection method according to claim 10, further comprising: determining a projection residual function based on the observation pixel coordinates and the projected pixel coordinates of the predetermined observation points in each of the frame images.

12. determining projected pixel coordinates of the preset observation point in each of the frame images based on camera parameters of the vehicle-mounted camera, the odometer frame information corresponding to each of the frame images, and world coordinates of the preset observation point, determining a projection relationship between a world coordinate system and a pixel coordinate system corresponding to each of the frame images based on a camera parameter of the vehicle-mounted camera and the odometer frame information corresponding to each of the frame images; and determining projected pixel coordinates of the preset observation point in each of the frame images based on the projection relationship and the world coordinates of the preset observation point.

13. The step of determining an imaging residual function based on the bounding box and the odometer frame information corresponding to each of the frame images includes: determining an observation pixel size of a preset observation line of the traffic sign in each of the frame images based on the bounding box corresponding to each of the frame images; determining an actual dimension of said preset observation line; determining an imaging pixel size of the preset observation line in each of the frame images based on a camera parameter of the vehicle-mounted camera, the odometer frame information corresponding to each of the frame images, and an actual size of the preset observation line; The traffic sign detection method according to claim 10, further comprising: determining an imaging residual function based on the observation pixel size and the imaging pixel size of the preset observation line in each of the frame images.

14. determining an imaging pixel size of the preset observation line in each of the frame images based on a camera parameter of the vehicle-mounted camera, the odometer frame information corresponding to each of the frame images, and an actual size of the preset observation line, determining a projection relationship between a world coordinate system and a pixel coordinate system corresponding to each of the frame images based on a camera parameter of the vehicle-mounted camera and the odometer frame information corresponding to each of the frame images; and determining an imaging pixel size of the preset observation line in each of the frame images based on the projection relationship and an actual size of the preset observation line.

15. determining a consistency residual function based on the bounding box and the odometer frame information corresponding to each of the frame images in response to the number of the traffic signs being greater than a preset threshold, determining a first traffic sign and a second traffic sign from the plurality of traffic signs in response to the number of traffic signs being greater than a preset threshold; determining a first camera coordinate of the first traffic sign and a second camera coordinate of the second traffic sign in each of the frame images based on the bounding box and the odometer frame information corresponding to each of the frame images; and determining a consistency residual function based on a first camera coordinate of the first traffic sign and a second camera coordinate of the second traffic sign in each of the frame images.

16. determining a real-time distance between the dashcam and the traffic sign based on the target position, determining a real-time position of the vehicle-mounted camera based on odometer frame information corresponding to each of the frame images; and determining a real-time distance between the dashcam and the traffic sign based on a real-time position of the dashcam and the target position.

17. A first determination module is used to determine an image frame sequence of a traffic sign collected by an on-board camera of a vehicle, and odometer frame information corresponding to each frame image in the image frame sequence; A second determination module is used to determine a residual function and an optimization model based on each of the frame images and odometer frame information corresponding to each of the frame images; a third determination module used for determining a target position of the traffic sign in a preset coordinate system based on the residual function and the optimization model; and a fourth determination module used for determining a target position and orientation of the traffic sign or a real-time distance between the vehicle-mounted camera and the traffic sign based on the target position.

18. A computer-readable storage medium storing a computer program used to execute the method for detecting a traffic sign according to any one of claims 1 to 16.

19. A processor; A memory adapted to store instructions executable by the processor; The processor is an electronic device used to read the instructions from the memory and execute the instructions to realize the traffic sign detection method according to any one of claims 1 to 16.

Images