Traffic sign recognition method and device

Abstract

The application discloses a traffic sign recognition method and device, and the method comprises the following steps: acquiring recognition information of a traffic sign, wherein the recognition information comprises three-dimensional coordinates of the traffic sign in a real space, a sign size, and relative position information of a camera; calculating two-dimensional coordinates of the traffic sign in a two-dimensional image according to the three-dimensional coordinates and the relative position information; calculating a selected region in which the traffic sign is imaged in the two-dimensional image according to the sign size and the two-dimensional coordinates; and performing image recognition on the traffic sign in the selected region to recognize the traffic sign, wherein the selected region is located in the two-dimensional image. The application can solve the technical problems of cost increase and difficulty in wide promotion in the prior art.

Description

Technical Field

[0001] This invention relates to the field of traffic technology, and in particular to a traffic sign recognition method and device. Background Technology

[0002] Currently, traffic light recognition technology, apart from vehicle-to-everything (V2X) systems, typically uses visible light (RGB) cameras for image recognition. However, for traffic light signs installed at large intersections, the cameras may be unable to capture images of the signs due to reasons such as the distance between the traffic light and the stop line, irregular intersection structures, or off-center installation locations, thus making it impossible to determine the current status of the traffic light.

[0003] Current technologies typically use wide-angle or telephoto cameras to capture images of traffic light signs. However, in practice, it has been found that while wide-angle cameras have a larger field of view and can capture traffic light signs at relatively angles, their images of distant signs are small, resulting in a limited effective viewing distance. This forces vehicles to approach or even drive to the intersection to recognize the traffic light, especially when the light is green, requiring them to slow down or stop, reducing driving efficiency and safety. Telephoto cameras, on the other hand, have a longer effective viewing distance and can capture clear images of distant traffic light signs, but their narrower field of view prevents them from capturing traffic light signs that are installed at an angle. Therefore, there is a fundamental contradiction between wide-angle and telephoto cameras, and no compromise solution exists.

[0004] Currently, to address the aforementioned issues, existing vehicles typically install both wide-angle and telephoto cameras simultaneously; or, multiple telephoto cameras at different angles. This undoubtedly increases vehicle costs and is unsuitable for widespread adoption and application. Therefore, there is an urgent need to propose a better traffic sign recognition solution. Summary of the Invention

[0005] This application provides a traffic sign recognition method and device, which solves the technical problems of increased cost and unsuitability for widespread promotion in the prior art.

[0006] On the one hand, this application provides a traffic sign recognition method through one embodiment of the application, the method comprising:

[0007] The identification information of the traffic sign is obtained, including the three-dimensional coordinates of the traffic sign in real space, the size of the sign, and the relative position information of the camera device;

[0008] Based on the three-dimensional coordinates and the relative position information, the two-dimensional coordinates of the traffic sign in the two-dimensional image are calculated. The two-dimensional image is the image obtained when the camera device captures the traffic sign.

[0009] Based on the sign size and the two-dimensional coordinates, the selected area where the traffic sign is imaged in the two-dimensional image is calculated;

[0010] The selected area is located in the two-dimensional image, and image recognition is performed on the traffic signs in the selected area to identify the traffic signs.

[0011] Optionally, the method further includes:

[0012] If the selected area is not located in the two-dimensional image, the angle of the gimbal is adjusted to adjust and update the relative position information of the camera device. The gimbal angle is the angle of the gimbal used to mount the camera device.

[0013] After updating the relative position information, the step of calculating the two-dimensional coordinates of the traffic sign in the two-dimensional image based on the three-dimensional coordinates and the relative position information is repeated.

[0014] Optionally, calculating the two-dimensional coordinates of the traffic sign in the two-dimensional image based on the three-dimensional coordinates and the relative position information includes:

[0015] Based on the three-dimensional coordinates and the vehicle position coordinates of the vehicle where the camera device is located, the first position coordinates of the traffic sign in the vehicle coordinate system are calculated;

[0016] The first position coordinates in the vehicle coordinate system are transformed to the camera coordinate system to obtain the second position coordinates;

[0017] The second position coordinates are projected and calculated based on the camera coordinate mapping matrix of the camera device to obtain the two-dimensional coordinates.

[0018] Optionally, calculating the selected area where the traffic sign is imaged in the two-dimensional image based on the sign size and the two-dimensional coordinates includes:

[0019] Based on the sign size and the camera parameters of the camera device, the area size of the traffic sign imaged in the two-dimensional image is calculated;

[0020] The selected region is calculated based on the region size and the two-dimensional coordinates.

[0021] Optionally, when the traffic sign is a traffic light sign, the step of performing image recognition on the traffic signs in the selected area to identify the traffic signs includes:

[0022] The color of each pixel in the selected area is calculated to obtain the color of each pixel in the selected area, wherein the color includes red, green and yellow;

[0023] The color of each pixel in the selected region is counted to obtain the number of pixels corresponding to each color;

[0024] The target color symbol indicated by the traffic light sign is determined based on the number of pixels of each color.

[0025] Optionally, the step of performing color calculation on each pixel in the selected region to obtain the color of each pixel in the selected region includes any one of the following:

[0026] If the red channel value of the target pixel exceeds the channel threshold corresponding to the green channel value and the blue channel value of the target pixel, then the color of the target pixel is determined to be red.

[0027] If the green channel value of the target pixel exceeds the channel threshold corresponding to the red channel value and the blue channel value of the target pixel, then the color of the target pixel is determined to be green.

[0028] If the red channel value and blue channel value of the target pixel both exceed the corresponding channel threshold, and the green channel value of the target pixel is less than the preset threshold, then the color of the target pixel is determined to be yellow.

[0029] The target pixel is any pixel in the selected region.

[0030] Optionally, before adjusting the gimbal angle, the method further includes:

[0031] The adjustment angle of the gimbal is calculated based on the first coordinate position of the traffic sign in the vehicle coordinate system and the relative position information.

[0032] The adjustment of the gimbal angle includes:

[0033] Adjust the gimbal according to the stated adjustment angle.

[0034] Optionally, calculating the adjustment angle of the gimbal based on the first coordinate position of the traffic sign in the vehicle coordinate system and the relative position information includes:

[0035] If the center position of the traffic sign moves into the two-dimensional image, the adjustment angle is calculated based on the first coordinate position of the traffic sign in the vehicle coordinate system, the relative position information, and the vertical and horizontal field of view of the camera device.

[0036] Optionally, calculating the adjustment angle of the gimbal based on the first coordinate position of the traffic sign in the vehicle coordinate system and the relative position information includes:

[0037] If all the traffic signs are moved into the two-dimensional image, the adjustment angle is calculated based on the first coordinate position of the traffic sign in the vehicle coordinate system, the relative position information, the vertical field of view and the horizontal field of view of the camera device, and the size of the sign.

[0038] On the other hand, this application provides a traffic sign recognition device through one embodiment of the application. The device includes an acquisition module, a calculation module, and a recognition module, wherein:

[0039] The acquisition module is used to acquire the identification information of the traffic sign, which includes the three-dimensional coordinates of the traffic sign in real space, the sign size, and the relative position information of the camera device.

[0040] The calculation module is used to calculate the two-dimensional coordinates of the traffic sign in the two-dimensional image based on the three-dimensional coordinates and the relative position information. The two-dimensional image is the image obtained when the camera device captures the traffic sign.

[0041] The calculation module is also used to calculate the selected area in the two-dimensional image where the traffic sign is imaged, based on the sign size and the two-dimensional coordinates.

[0042] The recognition module is used to perform image recognition on traffic signs in the selected area, where the selected area is located in the two-dimensional image, so as to identify the traffic signs.

[0043] For any content not introduced or described in the embodiments of this application, please refer to the relevant descriptions in the foregoing method embodiments; they will not be repeated here.

[0044] On the other hand, this application provides a terminal device through one embodiment of the present application. The terminal device includes: a processor, a memory, a communication interface, and a bus; the processor, the memory, and the communication interface are connected through the bus and complete communication with each other; the memory stores executable program code; the processor reads the executable program code stored in the memory to run a program corresponding to the executable program code, so as to execute the traffic sign recognition method as described above.

[0045] On the other hand, this application provides a computer-readable storage medium through one embodiment of the application, the computer-readable storage medium storing a program that, when the program is run on a terminal device, performs the traffic sign recognition method as described above.

[0046] One or more technical solutions provided in this application embodiment have at least the following technical effects or advantages: This application obtains the identification information of traffic signs, the identification information including the three-dimensional coordinates of the traffic sign in real space, the sign size, and the relative position information of the camera device; based on the three-dimensional coordinates and the relative position information, the two-dimensional coordinates of the traffic sign in a two-dimensional image are calculated, the two-dimensional image being the image obtained when the camera device captures the traffic sign; based on the sign size and the two-dimensional coordinates, a selected area of ​​the traffic sign image in the two-dimensional image is calculated; if the selected area is located in the two-dimensional image, image recognition is performed on the traffic sign in the selected area to identify the traffic sign. In the above solution, this application can determine whether the selected area of ​​the traffic sign is located in the two-dimensional image captured by the camera device based on the identification information of the traffic sign; and then accurately identify the traffic sign in the selected area, thus avoiding invalid identification of traffic signs when there are no traffic signs in the two-dimensional image, thereby improving the accuracy and convenience of traffic sign recognition. It also solves the technical problems of increased cost and unsuitability for widespread promotion existing in the prior art. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating a traffic sign recognition method provided in an embodiment of this application.

[0049] Figure 2 This is a schematic diagram of the real space coordinate system, vehicle coordinate system, camera coordinate system, image coordinate system, and their transformation relationship provided in the embodiments of this application.

[0050] Figure 3 This is a schematic diagram illustrating the composition and installation method of a traffic sign recognition device provided in an embodiment of this application.

[0051] Figure 4 This is a schematic diagram of a selected area in a two-dimensional image provided in an embodiment of this application.

[0052] Figures 5-6 These are schematic diagrams illustrating two possible gimbal angle adjustments provided in the embodiments of this application.

[0053] Figure 7 This is a schematic diagram of the structure of a traffic sign recognition device provided in an embodiment of this application.

[0054] Figure 8 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0055] This application provides a traffic sign recognition method and device, which solves the technical problems of increased cost and unsuitability for widespread promotion in the prior art.

[0056] The technical solution of this application embodiment is to solve the above-mentioned technical problems. The overall idea is as follows: Obtain the identification information of traffic signs, the identification information including the three-dimensional coordinates of the traffic signs in real space, the size of the signs, and the relative position information of the camera device; calculate the two-dimensional coordinates of the traffic signs in a two-dimensional image based on the three-dimensional coordinates and the relative position information, the two-dimensional image being the image obtained when the camera device captures the traffic signs; calculate the selected area of ​​the traffic signs in the two-dimensional image based on the size of the signs and the two-dimensional coordinates; if the selected area is located in the two-dimensional image, perform image recognition on the traffic signs in the selected area to identify the traffic signs.

[0057] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0058] First, it should be clarified that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0059] Please see Figure 1 This is a flowchart illustrating a traffic sign recognition method provided in an embodiment of this application. Figure 1 The method shown is applied to a terminal device, including but not limited to computers, vehicles, mobile phones, PDAs, personal computers, servers, or other devices with access control functions. The method includes the following implementation steps:

[0060] S101. Obtain the identification information of the traffic sign, the identification information including the three-dimensional coordinates of the traffic sign in real space, the size of the sign, and the installation position information of the camera device relative to the vehicle body.

[0061] The identification information described in this application is information used to identify the traffic sign, which may include, but is not limited to, the three-dimensional coordinates of the traffic sign in real space, the sign size, the relative position information of the camera device relative to the vehicle body, or other system-defined information. The sign size includes the length H of the traffic sign. target and width W target The relative position information includes, but is not limited to, the camera position coordinates (x, y) of the camera device relative to the vehicle coordinate system. camera y camera , z camera The rotation angles (R) of the camera device relative to the xyz axes of the vehicle coordinate system. camera P camera Y camera ( ), or other custom information. Where R, P, and Y are the rotation angles of the camera device relative to the vehicle coordinate system around the three directional axes X, Y, and Z. Optionally, this application can store the identification information in a corresponding storage device (e.g., a memory) so that the terminal device can read it at any time.

[0062] Please see Figure 2 The diagram illustrates the real space coordinate system, the vehicle coordinate system, the camera coordinate system, the image coordinate system, and their transformation relationships. Figure 3 This diagram illustrates the composition, installation method, and scenario of a vehicle capturing traffic signs using the traffic sign recognition device proposed in the embodiment. As shown, the vehicle is equipped with a positioning device and a camera. When driving through an intersection, the camera can capture images of the traffic signs installed at the intersection. The diagram uses a traffic light sign as an example, but this is not intended to limit the scope; the traffic signs involved in this application can also be turning signs or other traffic signs. The camera captures a two-dimensional image of the traffic sign. It should be noted that the description of the traffic sign location below refers specifically to the center position of the traffic sign. Figure 3 In this system, the camera device can be mounted on a gimbal, and the vehicle supports communication with external devices, such as computers. The computers can control the angle adjustment of the gimbal, thereby adjusting the relative position information of the camera device relative to the vehicle, such as the relative pitch angle and horizontal deflection angle of the camera device.

[0063] It should be noted that the camera device involved in this application may include, but is not limited to, a monocular camera, a binocular camera or a multi-camera, and the type of camera device is not limited, such as a telephoto camera or a wide-angle camera.

[0064] S102. Based on the three-dimensional coordinates and the relative position information, calculate the two-dimensional coordinates of the traffic sign in the two-dimensional image, where the two-dimensional image is the image obtained when the camera device captures the traffic sign.

[0065] In one specific embodiment, this application can calculate the first position coordinates of the traffic sign in the vehicle coordinate system based on the three-dimensional coordinates and the vehicle position coordinates of the vehicle where the camera device is located.

[0066] In specific implementation, this application can calculate the first position coordinates of the traffic sign in the vehicle coordinate system based on the three-dimensional coordinates of the traffic sign in real space, the vehicle position coordinates of the vehicle in real space, and the vehicle rotation matrix of the vehicle in real space. Specifically, for example, this application can use the following formula (1) to calculate the first position coordinates.

[0067]

[0068] Among them, (x t2v y t2v , z t2v (x) represents the first position coordinate of the traffic sign (specifically, the center position of the traffic sign) in the vehicle coordinate system. target y target , z target (x) represents the three-dimensional coordinates of the traffic sign in real space. vehicle y vehicle , z vehicle ) represents the vehicle's position coordinates.

[0069] T(R vehicle ,R vehicle ,Y vehicle R is the vehicle rotation matrix. vehicle P vehicle and Y vehicle These represent the rotation angles of the vehicle along the three directional axes in real space. Wherein, T(R) vehicle ,P vehicle ,Y vehicle Specifically, it can be represented by the following formula (2):

[0070]

[0071] In practical applications, x vehicle y vehicle z vehicle R vehicle P vehicle and Y vehicle All parameters are obtained by the vehicle through positioning equipment.

[0072] Next, the first position coordinates in the vehicle coordinate system are transformed to the camera coordinate system to obtain the second position coordinates. In specific implementation, this application can transform the first position coordinates from the vehicle coordinate system to the camera coordinate system based on the camera position coordinates of the camera device relative to the vehicle body and the camera rotation matrix of the camera device relative to the vehicle body, thereby obtaining the second position coordinates. Specifically, the second position coordinates can be calculated using, for example, the following formula (3).

[0073]

[0074] Among them, (x t2c y t2c , z t2c (x) represents the second position coordinate of the traffic sign in the camera coordinate system. camera y camera , z camera T(R) represents the camera position coordinates relative to the vehicle body. camera ,P camera ,Y camera R is the camera rotation matrix. camera P camera and Y camera These are the rotation angles of the camera device relative to the vehicle body along the three directional axes, respectively. Wherein, (x... camera y camera , z camera R camera P camera Y camera All of these are the mounting position information of the camera device relative to the vehicle body. This information has already been obtained in S101.

[0075] Finally, based on the camera coordinate mapping matrix of the camera device, the second position coordinates are projected and calculated to obtain the two-dimensional coordinates. In specific implementation, this application calculates the two-dimensional coordinates of the traffic sign in the two-dimensional image using the projection formula of the camera device. The specific calculation process is shown in the following formula (4):

[0076]

[0077] Among them, (x img ,y img ) represents the two-dimensional coordinates. T cam The camera coordinate mapping matrix can be expressed as shown in the following formula (5):

[0078]

[0079] Among them, dpm x and dpmy Res represents the resolution (i.e., pixel density) of the camera device on the X and Y axes, respectively, with units of pixels per meter. x and Res y These represent the maximum number of pixels on the X and Y axes of the camera device, respectively, which are the pixel dimensions of the two-dimensional image captured by the camera device.

[0080] S103. Based on the sign size and the two-dimensional coordinates, calculate the selected area where the traffic sign is imaged in the two-dimensional image.

[0081] In one specific embodiment, this application can calculate the area size of the traffic sign in the two-dimensional image based on the sign size and the camera parameters of the camera device.

[0082] In specific implementation, this application can calculate the area size based on the sign size, the camera parameters, and the coordinates of the traffic sign in the X-axis direction of the camera coordinate system (i.e., the X-direction coordinates in the second position coordinates). The camera parameters in this application are parameters used to describe the camera device, such as the camera focal length of the camera device. The specific calculation is shown in the following formula (6):

[0083]

[0084] Among them, (H) camera W camera ) represents the length and width of the area. camera H is the focal length of the camera device. target W target ) represents the length and width of the logo size.

[0085] Furthermore, this application calculates the selected region based on the region size and the two-dimensional coordinates. In specific implementation, this application can calculate the selected region based on the region size, the two-dimensional coordinates, and a preset region selection margin (also known as a preset coefficient), as shown in the following formula (7):

[0086]

[0087] Where (top, down, left, right) are the coordinates of the boundary points located on the four boundaries of the selected region. mar gin Choose a margin for the region, typically k mar gin >1. In k mar ginWhen the value is greater than 1, the selected area is larger than the imaging area of ​​the traffic sign in the two-dimensional image. The purpose is to ensure that, even with measurement calibration errors, the traffic sign still falls within the selected area of ​​the two-dimensional image when the camera takes a picture.

[0088] For example, please see Figure 4 This is a schematic diagram of a two-dimensional image including a selected region. For example... Figure 4 As shown, the selected area (top, down, left, right) and the two-dimensional coordinates of the traffic sign in the two-dimensional image are given. The figure shows the two-dimensional coordinates of the center position of the traffic sign in the two-dimensional image.

[0089] Furthermore, this application determines whether the selected area is located within the two-dimensional image captured by the camera device based on the boundary coordinates of the selected area of ​​the traffic sign. The determination method is as follows:

[0090] If top>0 and down <Res y And left > 0 and right <Res x Then it can be determined that the selected region is located within the two-dimensional image;

[0091] If top>Res y or down < 0 or left > Res x If right < 0, then it can be determined that the selected area is not within the two-dimensional image at all;

[0092] In other cases, the selected area is not entirely within the two-dimensional image; that is, part of the selected area is within the two-dimensional image and part is outside the two-dimensional image.

[0093] S104. When the selected area is located in the two-dimensional image, perform image recognition on the traffic signs in the selected area to identify the traffic signs.

[0094] This application can determine whether the selected area is completely located within the two-dimensional image. If so, this application can perform image recognition on traffic signs in the selected area to identify the traffic signs. Specific implementation details of the image recognition method are not described in detail here.

[0095] Taking a traffic light as an example, this application can calculate the color of each pixel in the selected area to obtain the color of each pixel in the selected area, where the colors include red, green, and yellow. Specifically, for each pixel in the selected area, the color can be calculated according to the following implementation method: Taking a target pixel in the selected area as an example, if this application determines that the red channel value of the target pixel exceeds a certain channel threshold corresponding to the green and blue channel values ​​of the target pixel, then this application can determine that the color of the target pixel is red. If this application determines that the green channel value of the target pixel exceeds a certain channel threshold corresponding to the red and blue channel values ​​of the target pixel, then this application can determine that the color of the target pixel is green. Conversely, if this application determines that the red and blue channel values ​​of the target pixel both exceed the corresponding certain channel thresholds, and the green channel value of the target pixel is less than a certain preset threshold, then this application can determine that the color of the target pixel is yellow.

[0096] Next, this application counts the color of each pixel in the selected region to obtain the number of pixels corresponding to each color. Finally, if the number of pixels of a certain color exceeds the number of pixels of the other two colors by a certain preset threshold T... light If the color is confirmed to be the target color indicated by the traffic light sign, then the identification fails and the target color status cannot be determined. Specifically, for example, if the number of red pixels exceeds the number of yellow and green pixels by a threshold T... light When the number of green pixels exceeds the number of yellow pixels and the number of red pixels by a threshold T, this application can determine that the target color currently indicated by the traffic light sign is red. light When the number of yellow pixels exceeds the number of green pixels and the number of red pixels by a threshold T, this application can determine that the target color currently indicated by the traffic light sign is green. light If the traffic light indicator is not met, this application can determine that the target color currently indicated by the traffic light is yellow. If none of the above conditions are met, the recognition fails, and this application cannot determine the target color status.

[0097] The following describes some optional embodiments related to this application.

[0098] In some alternative embodiments, when the selected area is not located in the two-dimensional image, the gimbal angle can be adjusted so that the selected area is located in the two-dimensional image. Understandably, adjusting the gimbal angle also synchronously adjusts the relative position information of the camera device.

[0099] When adjusting the pan-tilt angle, this application can simultaneously adjust and update the relative position information of the camera device. After updating the relative position information, steps S102-S104 can be repeated to re-identify traffic signs in the selected area.

[0100] In an optional embodiment, this application can calculate the adjustment angle (P) of the gimbal based on the first coordinate position of the traffic sign in the vehicle coordinate system and the relative position information. adjust ,Y adjust The gimbal is then adjusted according to the aforementioned adjustment angle. Wherein, P adjust Y represents the pitch angle of the gimbal. adjust The angle of orientation of the gimbal is also known as the horizontal deflection angle.

[0101] The following describes some specific embodiments related to calculating the adjustment angle of the gimbal.

[0102] In some embodiments, when the center position of the traffic sign is adjusted (or moved) to the center position of the two-dimensional image, the adjustment angle of the gimbal can be calculated using the following formula (8).

[0103]

[0104]

[0105] In some embodiments, when the center position of the traffic sign is adjusted (or moved) to the interior of the two-dimensional image, the adjustment angle can be calculated based on the first coordinate position of the traffic sign in the vehicle coordinate system, the relative position information, and the vertical and horizontal field of view of the camera device. Specifically, in this case, the minimum and maximum adjustment angles can be calculated using the following formulas (9) and (10). In other words, the adjustment angle fluctuates between the minimum and maximum adjustment angles.

[0106]

[0107]

[0108] Where, θ v and θ h These are the vertical field of view and the horizontal field of view of the camera device, respectively.

[0109]

[0110]

[0111] In some embodiments, when the selected area of ​​the traffic sign is moved entirely into the two-dimensional image, the application can calculate the adjustment angle based on the first coordinate position of the traffic sign in the vehicle coordinate system, the relative position information, the vertical field of view and horizontal field of view of the camera device, and the sign size. Specifically, in this case, the application can use the following formulas (11) and (12) to calculate the minimum and maximum adjustment angles. In other words, the adjustment angle fluctuates between the minimum and maximum adjustment angles.

[0112]

[0113]

[0114] Among them, H adjust and W adjust These are the adjustment length and adjustment width, determined according to the size of the mark, respectively.

[0115]

[0116]

[0117] Among them, H adjust and W adjust The following formulas (13) are shown respectively:

[0118]

[0119]

[0120] It should be noted that the gimbal described in this application can be adjusted between the maximum adjustment angle and the minimum adjustment angle, and (P adjust ,Y adjust Adjustments should be made within the range supported by the gimbal's mechanical structure.

[0121] After obtaining the adjustment angle of the gimbal, this application can synchronously update the relative position information of the camera device. Then, steps S102-S104 above are re-executed, wherein the specific calculations involved in formula (3) above are also synchronously updated, as shown in formula (14) below:

[0122]

[0123] For example, please see also Figure 5 and Figure 6 These are schematic diagrams illustrating two possible gimbal angle adjustments provided in the embodiments of this application. For example... Figure 5 The image shows the gimbal performing a Y-axis movement in the horizontal direction. adjust Angle adjustment. For example...Figure 6 The image shows the gimbal performing P movements in the pitch direction. adjust Angle adjustment.

[0124] By implementing the embodiments of this application, this application obtains the identification information of traffic signs, including the three-dimensional coordinates of the traffic sign in real space, the sign size, and the relative position information of the camera device; based on the three-dimensional coordinates and the relative position information, the two-dimensional coordinates of the traffic sign in a two-dimensional image are calculated, where the two-dimensional image is the image obtained when the camera device captures the traffic sign; based on the sign size and the two-dimensional coordinates, a selected area of ​​the traffic sign image in the two-dimensional image is calculated; since the selected area is located in the two-dimensional image, image recognition is performed on the traffic sign in the selected area to identify the traffic sign. In the above solution, this application can determine whether the selected area of ​​the traffic sign is located within the two-dimensional image captured by the camera device based on the identification information of the traffic sign; thus, it can accurately identify the traffic sign within the selected area, thereby avoiding invalid identification of the traffic sign when there is no traffic sign in the selected area, thereby improving the accuracy and convenience of traffic sign recognition. It also solves the technical problems of increased cost and unsuitability for widespread promotion existing in the prior art.

[0125] Based on the same inventive concept, another embodiment of this application provides an apparatus and terminal device for implementing the traffic sign recognition method described in the embodiments of this application.

[0126] Please see Figure 7 This is a schematic diagram of the structure of a traffic sign recognition device provided in an embodiment of this application. Figure 7 The device shown includes an acquisition module 701, a calculation module 702, and an identification module 703, wherein:

[0127] The acquisition module 701 is used to acquire the identification information of the traffic sign, which includes the three-dimensional coordinates of the traffic sign in real space, the sign size, and the relative position information of the camera device.

[0128] The calculation module 702 is used to calculate the two-dimensional coordinates of the traffic sign in the two-dimensional image based on the three-dimensional coordinates and the relative position information. The two-dimensional image is the image obtained when the camera device captures the traffic sign.

[0129] The calculation module 702 is further configured to calculate, based on the sign size and the two-dimensional coordinates, the selected area in which the traffic sign is imaged in the two-dimensional image;

[0130] The recognition module 703 is used to perform image recognition on traffic signs in the selected area, where the selected area is located in the two-dimensional image, so as to identify the traffic signs.

[0131] Optionally, the device further includes a processing module 704, wherein:

[0132] The processing module 704 is used to adjust the gimbal angle when the selected area is not located in the two-dimensional image, so as to adjust and update the relative position information of the camera device, wherein the gimbal angle is the angle of the gimbal used to mount the camera device; after updating the relative position information, the step of calculating the two-dimensional coordinates of the traffic sign in the two-dimensional image based on the three-dimensional coordinates and the relative position information is repeated.

[0133] Optionally, the calculation module 702 is specifically used for:

[0134] Based on the three-dimensional coordinates and the vehicle position coordinates of the vehicle where the camera device is located, the first position coordinates of the traffic sign in the vehicle coordinate system are calculated;

[0135] The first position coordinates in the vehicle coordinate system are transformed to the camera coordinate system to obtain the second position coordinates;

[0136] The second position coordinates are projected and calculated based on the camera coordinate mapping matrix of the camera device to obtain the two-dimensional coordinates.

[0137] Optionally, the calculation module 702 is specifically used for:

[0138] Based on the sign size and the camera parameters of the camera device, the area size of the traffic sign imaged in the two-dimensional image is calculated;

[0139] The selected region is calculated based on the region size and the two-dimensional coordinates.

[0140] Optionally, the traffic sign is a traffic light sign, and the recognition module 703 is specifically used for:

[0141] The color of each pixel in the selected area is calculated to obtain the color of each pixel in the selected area, wherein the color includes red, green and yellow;

[0142] The color of each pixel in the selected region is counted to obtain the number of pixels corresponding to each color;

[0143] The target color symbol indicated by the traffic light sign is determined based on the number of pixels of each color.

[0144] Optionally, the identification module 703 is specifically configured to perform any of the following:

[0145] If the red channel value of the target pixel exceeds the channel threshold corresponding to the green channel value and the blue channel value of the target pixel, then the color of the target pixel is determined to be red.

[0146] If the green channel value of the target pixel exceeds the channel threshold corresponding to the red channel value and the blue channel value of the target pixel, then the color of the target pixel is determined to be green.

[0147] If the red channel value and blue channel value of the target pixel both exceed the corresponding channel threshold, and the green channel value of the target pixel is less than the preset threshold, then the color of the target pixel is determined to be yellow.

[0148] The target pixel is any pixel in the selected region.

[0149] Optionally, the processing module 704 is further configured to calculate the adjustment angle of the gimbal based on the first coordinate position of the traffic sign in the vehicle coordinate system and the relative position information;

[0150] The processing module 704 is specifically used to adjust the gimbal according to the adjustment angle.

[0151] Optionally, the processing module 704 is specifically used for:

[0152] If the center position of the traffic sign moves into the two-dimensional image, the adjustment angle is calculated based on the first coordinate position of the traffic sign in the vehicle coordinate system, the relative position information, and the vertical and horizontal field of view of the camera device.

[0153] Optionally, the processing module 704 is specifically used for:

[0154] If the selected area of ​​the traffic sign is moved entirely into the two-dimensional image, the adjustment angle is calculated based on the first coordinate position of the traffic sign in the vehicle coordinate system, the relative position information, the vertical field of view and the horizontal field of view of the camera device, and the size of the sign.

[0155] Please also refer to Figure 8, which is a structural schematic diagram of a terminal device provided in an embodiment of this application. Figure 8 The terminal device 80 shown includes at least one processor 801, a communication interface 802, a user interface 803, and a memory 804. The processor 801, communication interface 802, user interface 803, and memory 804 can be connected via a bus or other means; this embodiment of the invention takes connection via bus 805 as an example.

[0156] Processor 801 can be a general-purpose processor, such as a central processing unit (CPU).

[0157] The communication interface 802 can be a wired interface (e.g., an Ethernet interface) or a wireless interface (e.g., a cellular network interface or a wireless LAN interface) for communicating with other terminals or websites. In this embodiment of the invention, the communication interface 802 is specifically used to acquire identification information.

[0158] The user interface 803 can specifically be a touch panel, including a touch screen and a touch screen display, used to detect operation commands on the touch panel. The user interface 803 can also be a physical button or a mouse. The user interface 803 can also be a display screen, used to output and display images or data.

[0159] The memory 804 may include volatile memory, such as random access memory (RAM); the memory may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); the memory 804 may also include combinations of the above types of memory. The memory 804 is used to store a set of program code, and the processor 801 is used to call the program code stored in the memory 804 to execute part or all of the content of the traffic sign recognition method described above, which is not limited in this application.

[0160] Since the terminal device described in this embodiment is the terminal device used to implement the information processing method in the embodiments of this application, those skilled in the art can understand the specific implementation method and various variations of the terminal device in this embodiment based on the information processing method described in the embodiments of this application. Therefore, how the terminal device implements the method in the embodiments of this application will not be described in detail here. Any terminal device used by those skilled in the art to implement the information processing method in the embodiments of this application falls within the scope of protection of this application.

[0161] One or more technical solutions provided in this application embodiment have at least the following technical effects or advantages: This application obtains the identification information of traffic signs, the identification information including the three-dimensional coordinates of the traffic sign in real space, the sign size, and the relative position information of the camera device; based on the three-dimensional coordinates and the relative position information, the two-dimensional coordinates of the traffic sign in a two-dimensional image are calculated, the two-dimensional image being the image obtained when the camera device captures the traffic sign; based on the sign size and the two-dimensional coordinates, a selected area of ​​the traffic sign image in the two-dimensional image is calculated; if the selected area is located in the two-dimensional image, image recognition is performed on the traffic sign in the selected area to identify the traffic sign. In the above solution, this application can determine whether the selected area of ​​the traffic sign is located within the two-dimensional image captured by the camera device based on the identification information of the traffic sign; and then accurately identify the traffic sign within the selected area, thus avoiding invalid identification of traffic signs when there are no traffic signs in the two-dimensional image, thereby improving the accuracy and convenience of traffic sign recognition. It also solves the technical problems of increased cost and unsuitability for widespread promotion existing in the prior art.

[0162] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0164] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0165] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0166] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0167] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A traffic sign recognition method, characterized in that, The method includes: The identification information of the traffic sign is obtained, including the three-dimensional coordinates of the traffic sign in real space, the size of the sign, and the relative position information of the camera device; Based on the three-dimensional coordinates and the relative position information, the two-dimensional coordinates of the traffic sign in the two-dimensional image are calculated. The two-dimensional image is the image obtained when the camera device captures the traffic sign. Based on the sign size and the two-dimensional coordinates, the selected area where the traffic sign is imaged in the two-dimensional image is calculated; The selected area is located in the two-dimensional image, and image recognition is performed on the traffic signs in the selected area to identify the traffic signs; If the selected area is not located in the two-dimensional image, the gimbal angle is adjusted to adjust and update the relative position information of the camera device. The gimbal angle is the angle of the gimbal used to mount the camera device. After updating the relative position information, repeat the step of calculating the two-dimensional coordinates of the traffic sign in the two-dimensional image based on the three-dimensional coordinates and the relative position information; The step of calculating the selected area of ​​the traffic sign in the two-dimensional image based on the sign size and the two-dimensional coordinates includes: calculating the area size of the traffic sign in the two-dimensional image based on the sign size, the camera parameters of the camera device, and the coordinates of the traffic sign in the X-axis direction of the camera coordinate system; and calculating the selected area based on the area size, the two-dimensional coordinates, and a preset coefficient. The formula for calculating the size of the region is as follows: ; in,( , ) represents the length and width of the region. H is the focal length of the camera device. target W target ) represents the length and width of the logo size, the Let X be the coordinate of the traffic sign in the X-axis direction of the camera coordinate system; The calculation formula for the selected region is as follows: ； Where (top, down, left, right) are the coordinates of the boundary points located on the four boundaries of the selected area. For the preset coefficient, ( () represents two-dimensional coordinates.

2. The method according to claim 1, characterized in that, The step of calculating the two-dimensional coordinates of the traffic sign in the two-dimensional image based on the three-dimensional coordinates and the relative position information includes: Based on the three-dimensional coordinates and the vehicle position coordinates of the vehicle where the camera device is located, the first position coordinates of the traffic sign in the vehicle coordinate system are calculated; The first position coordinates in the vehicle coordinate system are transformed to the camera coordinate system to obtain the second position coordinates; The second position coordinates are projected and calculated based on the camera coordinate mapping matrix of the camera device to obtain the two-dimensional coordinates.

3. The method according to claim 1, characterized in that, The traffic sign is a traffic light sign, and the step of performing image recognition on the traffic signs in the selected area to identify the traffic signs includes: The color of each pixel in the selected area is calculated to obtain the color of each pixel in the selected area, wherein the color includes red, green and yellow; The color of each pixel in the selected region is counted to obtain the number of pixels corresponding to each color; The target color symbol indicated by the traffic light sign is determined based on the number of pixels of each color.

4. The method according to claim 3, characterized in that, The color calculation for each pixel in the selected region, resulting in the color of each pixel in the selected region, includes any one of the following: If the red channel value of the target pixel exceeds the channel threshold corresponding to the green channel value and the blue channel value of the target pixel, then the color of the target pixel is determined to be red. If the green channel value of the target pixel exceeds the channel threshold corresponding to the red channel value and the blue channel value of the target pixel, then the color of the target pixel is determined to be green. If the red channel value and blue channel value of the target pixel both exceed the corresponding channel threshold, and the green channel value of the target pixel is less than the preset threshold, then the color of the target pixel is determined to be yellow. The target pixel is any pixel in the selected region.

5. The method according to claim 1, characterized in that, Before adjusting the gimbal angle, the method further includes: The adjustment angle of the gimbal is calculated based on the first coordinate position of the traffic sign in the vehicle coordinate system and the relative position information. The adjustment of the gimbal angle includes: Adjust the gimbal according to the stated adjustment angle.

6. The method according to claim 5, characterized in that, The step of calculating the adjustment angle of the gimbal based on the first coordinate position of the traffic sign in the vehicle coordinate system and the relative position information includes: If the center position of the traffic sign moves into the two-dimensional image, the adjustment angle is calculated based on the first coordinate position of the traffic sign in the vehicle coordinate system, the relative position information, and the vertical and horizontal field of view of the camera device.

7. The method according to claim 5, characterized in that, The step of calculating the adjustment angle of the gimbal based on the first coordinate position of the traffic sign in the vehicle coordinate system and the relative position information includes: If all the traffic signs are moved into the two-dimensional image, the adjustment angle is calculated based on the first coordinate position of the traffic signs in the vehicle coordinate system, the relative position information, the vertical field of view and the horizontal field of view of the camera device, and the size of the signs.

8. A terminal device, characterized in that, The terminal device includes: a processor, a memory, a communication interface, and a bus; the processor, the memory, and the communication interface are connected through the bus and communicate with each other; the memory stores executable program code; the processor reads the executable program code stored in the memory to run a program corresponding to the executable program code, so as to execute the traffic sign recognition method as described in any one of claims 1-7 above.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when run on a terminal device, performs the traffic sign recognition method as described in any one of claims 1-7.

Images