Target extraction method and device, computer equipment, medium and program product
By dedistorting and inverse perspective transformation processing of roadside camera images, the target deformation problem caused by perspective effect is solved, accurate extraction of remote targets is achieved, and the quality of electronic maps is improved.
Patent Information
- Application Number
- CN202410850806.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-27
- Publication Date
- 2024-11-08
AI Technical Summary
In the prior art, the perspective effect generated by roadside camera imaging causes the remote target to occupy a small pixel range in the image and the deformation is large, resulting in missed and misdetection of the remote target by the deep learning model, affecting the quality of the electronic map.
By performing dedistortion processing and inverse perspective transformation processing on the images collected by the roadside camera, dedistortion images and inverse perspective transformation images are generated to eliminate changes in the shape, position and size of the target, thereby improving the accuracy of target extraction.
More accurate extraction of remote targets is achieved, the quality and accuracy of electronic maps are improved, and missed and false detection rates are reduced.
Smart Images

Figure CN118918562A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a target extraction method, device, computer equipment, medium and program product. Background Art
[0002] With the vigorous development of the intelligent driving field, the demand for updating and upgrading electronic map data is becoming increasingly strong. How to use roadside cameras for local mapping has become an indispensable technical means in crowd-source updating.
[0003] In related technologies, due to the perspective effect produced by roadside camera imaging, distant targets occupy a small pixel range on the image frame and have large deformations. When using deep learning models to perform target segmentation and extraction on images taken by roadside cameras, it is easy to cause missed detections and false detections of distant targets, ultimately affecting the quality of electronic maps.
[0004] Therefore, how to extract remote targets more accurately is a technical problem that needs to be solved urgently. Summary of the invention
[0005] Based on this, it is necessary to provide a target extraction method, device, computer equipment, medium and program product that can more accurately extract remote targets in response to the above technical problems.
[0006] In a first aspect, the present application provides a target extraction method, comprising:
[0007] When generating a map update instruction, obtaining an image to be identified, where the image to be identified is an image captured by a roadside camera;
[0008] Performing dedistortion processing on the image to be identified to obtain a dedistorted image;
[0009] Performing inverse perspective transformation processing on the dedistorted image based on regional point cloud data corresponding to the dedistorted image to obtain an inverse perspective transformed image of the dedistorted image;
[0010] The inverse perspective transformed image is input into a trained target segmentation model to obtain various road targets.
[0011] In one embodiment, performing dedistortion processing on the image to be identified to obtain a dedistorted image includes:
[0012] Determining optical parameters of the image to be recognized, wherein the optical parameters include a principal point, a focal length, and a distortion coefficient;
[0013] Dedistortion processing is performed on the image to be recognized based on the optical parameters to obtain a dedistorted image.
[0014] In one of the embodiments, performing inverse perspective transformation processing on the dedistorted image based on the regional point cloud data corresponding to the dedistorted image to obtain an inverse perspective transformed image of the dedistorted image includes:
[0015] The dedistorted image is fused with the corresponding regional point cloud data to obtain a transformation matrix corresponding to the dedistorted image, wherein the regional point cloud data is the point cloud data corresponding to the region captured by the image to be identified, and the transformation matrix represents the rotation and translation relationship between the roadside camera coordinate system and the road surface coordinate system;
[0016] The dedistorted image is projected from the roadside camera coordinate system to a ground coordinate system based on the transformation matrix to obtain an inverse perspective transformed image of the dedistorted image.
[0017] In one embodiment, the transformation matrix includes a rotation and translation relationship between a roadside camera coordinate system and a Gaussian coordinate system, and a rotation and translation relationship between a Gaussian coordinate system and a road surface coordinate system. Based on the transformation matrix, the dedistorted image is projected from the roadside camera coordinate system to a ground coordinate system to obtain an inverse perspective transformation image of the dedistorted image, including:
[0018] Based on the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system, projecting the dedistorted image into the Gaussian coordinate system to obtain a projection of the dedistorted image in the Gaussian coordinate system;
[0019] Based on the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system, the projection of the dedistorted image in the Gaussian coordinate system is mapped to the road surface coordinate system to obtain an inverse perspective transformation image of the dedistorted image.
[0020] In one embodiment, fusing the dedistorted image with the corresponding regional point cloud data to obtain a transformation matrix corresponding to the dedistorted image includes:
[0021] Determining a number of control points in the regional point cloud data;
[0022] Performing corner point recognition in the dedistorted image to determine a number of key points;
[0023] A transformation matrix is determined based on the correspondence between the control points and the key points.
[0024] In one embodiment, the training step of the target segmentation model includes:
[0025] Acquire multiple training images, where the training images are images captured by multiple roadside cameras;
[0026] Performing dedistortion processing on each of the training images to obtain a dedistorted image of each of the training images;
[0027] Based on the regional point cloud data corresponding to the dedistorted image, the dedistorted image is subjected to inverse perspective transformation processing to obtain an inverse perspective transformed image of the dedistorted image;
[0028] The inverse perspective transformed image of each dedistorted image is annotated, and each annotated inverse perspective transformed image is used as a training sample to train the initial model until the preset training conditions are met to obtain a trained target segmentation image.
[0029] In one embodiment, before acquiring the image to be recognized, the method further includes:
[0030] For each roadside camera, obtaining a training image taken by the roadside camera;
[0031] Extracting elements from the training image to obtain road surface elements;
[0032] Acquire original road surface elements, where the original road surface elements are road surface elements obtained by extracting features from a training image last taken by the roadside camera;
[0033] Comparing the road surface elements with the original road surface elements for similarity;
[0034] Based on the comparison result, it is determined whether to generate a map update instruction.
[0035] In one embodiment, determining whether to generate a map update instruction based on the comparison result includes:
[0036] If the comparison result shows that the similarity is greater than or equal to the set threshold, no map update instruction is generated;
[0037] If the comparison result is that the similarity is less than the set threshold, the position information of the roadside camera is obtained, and a map update instruction is generated based on the position information, and the map update instruction is used to instruct the point cloud collection vehicle to collect regional point cloud data based on the position information.
[0038] In a second aspect, the present application also provides a target extraction device, comprising:
[0039] An image acquisition module to be identified, used to acquire an image to be identified when generating a map update instruction, wherein the image to be identified is an image captured by a roadside camera;
[0040] A dedistortion module, used for performing dedistortion processing on the image to be identified to obtain a dedistorted image;
[0041] An inverse perspective transformation module, configured to perform inverse perspective transformation processing on the dedistorted image based on regional point cloud data corresponding to the dedistorted image, so as to obtain an inverse perspective transformed image of the dedistorted image;
[0042] The segmentation module is used to input the inverse perspective transformed image into a trained target segmentation model to obtain various road targets.
[0043] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the program steps of a target extraction method as described in the first aspect above are implemented.
[0044] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the program steps of a target extraction method as described in the first aspect above are implemented.
[0045] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the program steps of a target extraction method as described in the first aspect above.
[0046] The above-mentioned target extraction method, device, computer equipment, medium and program product perform dedistortion processing on the image to be identified to obtain a dedistorted image that can eliminate the change in the shape of the target, and then perform inverse perspective transformation processing on the dedistorted image to obtain an image from a bird's-eye view. From the bird's-eye view, each target is in the same plane. Therefore, the inverse perspective transformation image can eliminate the change in the shape, position, size and proportion of the remote target in the image to be identified taken by the roadside camera. Furthermore, by extracting the inverse perspective transformation image through the target segmentation model, the actual position, shape and size relationship of each road target in the image can be extracted more accurately. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0048] Figure 1 An application environment diagram of a target extraction method in an embodiment;
[0049] Figure 2 A schematic diagram of a target extraction method in one embodiment;
[0050] Figure 3is a schematic diagram of a flow chart of an inverse perspective transformation step in one embodiment;
[0051] Figure 4 is a schematic diagram of regional point cloud data and corresponding points in a dedistorted image in one embodiment;
[0052] Figure 5 is a schematic diagram of a process for determining a transformation matrix in another embodiment;
[0053] Figure 6 is a schematic diagram of an inverse perspective transformed image and a dedistorted image in one embodiment;
[0054] Figure 7 A schematic diagram of a target segmentation model training process in one embodiment;
[0055] Figure 8 is a structural block diagram of a target extraction device in one embodiment;
[0056] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0058] The target extraction method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the roadside camera 102 communicates with the computer device 104 through the network. The data storage system can store data that the computer device 104 needs to process. The data storage system can be integrated on the computer device 104, or it can be placed on the cloud or other network servers. Among them, the computer device 104 can be implemented with an independent server or a server cluster composed of multiple servers. The roadside camera 102 can actively send the collected images to the computer device 104, and the roadside camera 102 can also collect images in response to the image collection instruction sent by the computer device 104 and send them to the computer device 104.
[0059] In an exemplary embodiment, Figure 2 As shown, a target extraction method is provided, which is applied to Figure 1 The computer device 104 in the embodiment is used as an example for explanation. In the embodiment of the present application, the computer device is used as a server as an example for explanation. The target extraction method of the embodiment of the present application includes the following steps 10 to 40, wherein:
[0060] Step 10: When generating a map update instruction, obtain an image to be identified, where the image to be identified is an image captured by a roadside camera.
[0061] For the embodiment of the present application, when generating a map update instruction, a point cloud acquisition vehicle is required to acquire the specified area to obtain the regional point cloud data of the area. The image to be identified is an image taken by a roadside camera in the specified area for the specified area. Among them, the roadside camera is generally set on the truss, street light pole or electric pole on both sides of the road, and the roadside camera is generally set with a fixed viewing angle to monitor road traffic in a specified area. Therefore, there is a certain angle between the image taken by the roadside camera and the horizontal plane.
[0062] Step 20: Dedistort the image to be identified to obtain a dedistorted image.
[0063] Specifically, since general optical system cameras form images based on light, there will be a perspective effect. Therefore, targets that are far away from the shooting end will show certain changes in the captured image, which are specifically manifested as changes in position and shape, as well as changes in the ratio of size in the image to actual size. For example, a target A is set at a straight-line distance of 3 meters on one side of the shooting point, and a target B is set at 5 meters on the same side of the shooting point. Target A is exactly the same as target B. In the captured image, the size of target A is larger than that of target B. For another example, if two parallel lines are set in reality, the far ends of the two parallel lines will intersect and gradually overlap in the image. Therefore, in the image taken by the roadside camera, the pixel range occupied by the far-end target is small and the deformation is large.
[0064] Specifically, when dedistorting an image, it can be achieved through a trained image correction model or through related program software, such as The image dedistortion function in the image is used to achieve this. Of course, the image to be identified can also be dedistorted by other methods. No specific limitation is made in the embodiments of the present application, as long as it is easy to obtain a dedistorted image that represents the actual position and shape relationship of the target in the image more accurately.
[0065] Step 30: Perform inverse perspective transformation processing on the dedistorted image based on the regional point cloud data corresponding to the dedistorted image to obtain an inverse perspective transformed image of the dedistorted image.
[0066] For the embodiment of the present application, the regional point cloud data and the image to be identified are both collected for the same designated area, and therefore have a temporal and spatial correlation. Specifically, the intrinsic and extrinsic parameters of the roadside camera are used to eliminate the interference and errors of the perspective effect on the image detection and recognition tasks, thereby obtaining an inverse perspective transformation image; wherein the intrinsic parameters represent the inherent characteristics of the camera itself, such as focal length; and the extrinsic parameters represent the position and direction of the camera in the world coordinate system. The inverse perspective transformation image shows each target in the original image (image to be identified) from a bird's-eye view. From a bird's-eye view, each target is in the same plane, so the inverse perspective transformation image can eliminate changes in the shape, position, and proportion of distant targets in the image to be identified.
[0067] Step 40: Input the inverse perspective transformed image into the trained target segmentation model to obtain various road targets.
[0068] For the embodiment of the present application, the inverse perspective transformation image is a trained image that can segment the calibrated targets existing in the target, and then obtain the coordinates of each road surface. Among them, the road surface targets include road traffic signs, such as zebra crossings, driving direction arrows, lane lines, and other identification elements that can indicate traffic. Specifically, the output result of the target segmentation model can be a matrix representing the coordinates of each pixel point in the image and the road surface coordinate type to which it belongs, or it can be the position parameters corresponding to each road surface coordinate, or it can be a set of pixel points representing each road surface target. Among them, the position parameters can be in the image coordinate system or in the world coordinate system.
[0069] Furthermore, the pixel point set of each road target outputted by the target segmentation model is three-dimensionally reconstructed to obtain the world coordinate system of the characteristic object and complete the extraction of the road target.
[0070] In the embodiment of the present application, the image to be identified is subjected to dedistortion processing to obtain a dedistorted image that can eliminate the change in the target shape, and then the dedistorted image is subjected to inverse perspective transformation processing to obtain an image from a bird's-eye view. From a bird's-eye view, each target is in the same plane. Therefore, the inverse perspective transformation image can eliminate the change in the shape, position, size and proportion of the remote target in the image to be identified taken by the roadside camera. Furthermore, the inverse perspective transformation image is extracted through the target segmentation model, and the actual position, shape and size relationship of each road target in the image can be extracted more accurately.
[0071] Specifically, before step 201, the method further includes the step of generating a map update instruction. In practice, there will be multiple roadside cameras that can communicate with the server, and each roadside camera is set in a designated area for monitoring road traffic in the corresponding designated area and can collect images of the corresponding area. For each roadside camera, it includes: obtaining a training image taken by the roadside camera, and then extracting elements from the training image to obtain road surface elements; obtaining original road surface elements, which are road surface elements obtained by extracting features from the training image taken by the roadside camera last time; comparing the road surface elements with the original road surface elements for similarity; and determining whether to generate a map update instruction based on the comparison result.
[0072] Specifically, the roadside camera can be set to collect surveillance video of the corresponding area in real time, and the server obtains a frame image in the surveillance video as a training image at regular intervals; or the server can send a training image acquisition instruction to a roadside camera, so that the roadside camera takes an image of the corresponding area and sends it to the server. Among them, road surface elements include road traffic signs, such as zebra crossings, driving direction arrows, lane lines, and other identification elements that can indicate traffic. Since each roadside camera is fixed, the angle at which it collects training images is also fixed; road surface elements are usually fixed. Usually, even if the road surface elements are modified and adjusted, such as changing the straight arrow to a left turn and straight arrow, it is likely to be modified at the position of the original straight arrow. Therefore, target detection can be performed on fixed positions in the training image to obtain various road surface elements. The training image can also be input into a trained target segmentation or target detection model to determine the various road surface elements in the training image and their corresponding positions in the image.
[0073] Generally, the electronic map needs to be updated only when it is determined that the ground elements have changed. In the embodiment of the present application, for each roadside camera, when the detection corresponding to the roadside camera is obtained, each road surface element in the training image is extracted, and then the extracted road surface elements are compared with the original road surface elements at the corresponding positions for similarity, and then it is determined whether a map update instruction needs to be generated based on the comparison result.
[0074] Specifically, whether a map update instruction needs to be generated is determined based on the comparison result, including: if the comparison result between any road surface element and the corresponding original road surface element is that the similarity is greater than or equal to a set threshold, then no map update instruction is generated; if the comparison result between any road surface element and the corresponding original road surface element is that the similarity is less than the set threshold, then the position information of the roadside camera is obtained, and a map update instruction is generated based on the position information, and the map update instruction is used to instruct the point cloud collection vehicle to collect regional point cloud data based on the position information.
[0075] Furthermore, in step 20, dedistortion processing is performed on the image to be identified to obtain a dedistorted image, which may specifically include: determining optical parameters of the image to be identified, and dedistortion processing is performed on the image to be identified based on the optical parameters to obtain a dedistorted image.
[0076] Specifically, the optical parameters are intrinsic parameters, and the optical parameters include the principal point, focal length, and distortion coefficient when the roadside camera captures the image to be identified.
[0077] In one embodiment of the present application, if the image attributes of the image to be identified taken by the roadside camera contain optical parameters, the image attributes of the image to be identified can be directly read to obtain the optical parameters.
[0078] In another embodiment of the present application, a black and white checkerboard object or image can be placed in the collection area of the roadside camera, and then the roadside camera can be controlled to collect images. At this time, the image to be identified collected by the roadside camera includes a black and white checkerboard target, and then the image is imported into MATLAB (Matrix Laboratory) software to obtain the optical parameters of the roadside camera.
[0079] After determining the optical parameters, the image to be identified and the optical parameters can be input into the OpenCV (Open Source Computer Vision Library) software respectively, and the cv2.undistort() function can be used to remove the distortion, thereby obtaining the dedistorted image of the image to be identified.
[0080] Further, refer to Figure 3 , step 30, may specifically include step 31 and step 32, wherein:
[0081] Step 31: fuse the dedistorted image with the corresponding regional point cloud data to obtain a transformation matrix corresponding to the dedistorted image. The regional point cloud data is the point cloud data corresponding to the area captured by the image to be identified. The transformation matrix represents the rotation and translation relationship between the roadside camera coordinate system and the road surface coordinate system.
[0082] Specifically, the regional point cloud data and the dedistorted image of the image to be identified represent the setting relationship and position relationship of the same target in the same area, but the regional point cloud data represents the position, shape and size relationship of each target in the road coordinate system, and the dedistorted image corresponds to the position, shape and size relationship of each target in the two-dimensional perspective in the roadside camera's coordinate system. By establishing a corresponding relationship between each road target in the point cloud data and each road target in the dedistorted image, the transformation and translation relationship between the road coordinate system and the roadside camera's coordinate system, that is, the transformation matrix, can be determined, thereby realizing the fusion of the regional point cloud data and the dedistorted image.
[0083] Step 32: Project the dedistorted image from the roadside camera coordinate system to the ground coordinate system based on the transformation matrix to obtain an inverse perspective transformed image of the dedistorted image.
[0084] By projecting each road surface coordinate in the dedistorted image into the road surface coordinate system, the size, shape and position relationship of each road surface target in the bird's-eye view can be obtained, that is, the inverse perspective transformation image (IPM image).
[0085] Specifically, refer to Figure 4 , step 31 may specifically include: determining a number of control points in the regional point cloud data; performing corner point recognition in the dedistorted image to determine a number of key points; and determining a transformation matrix based on the correspondence between the control points and the key points.
[0086] Figure 4 The correspondence relationship characterizes the road targets in the regional point cloud data set and the road targets in the dedistorted image, and more specifically, the correspondence relationship between the key points in the regional point cloud data and the corner points in the dedistorted image. Due to the randomness of point cloud data acquisition, the targets contained in the regional point cloud data should be larger than the targets contained in the dedistorted image. Therefore, the targets in the point cloud data cannot directly establish a correspondence relationship with the targets in the dedistorted image. Specifically, several control points selected in the point cloud can be selected by the user; or the corner points of zebra crossings and arrows in the regional point cloud data can be automatically identified as control points based on the point cloud parsing software. Furthermore, the corner points of the dedistorted image are identified to determine the corner points of the road targets, and then the corner points are calibrated and displayed. By establishing the correspondence between the corner points and the control points, the transformation matrix can be determined.
[0087] Since the coordinate system of the roadside camera and the road surface coordinate system cannot be converted directly, in order to more accurately determine the conversion relationship between the two coordinate systems, a Gaussian coordinate system is introduced in the embodiment of the present application to perform relay conversion. Therefore, the transformation matrix specifically includes the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system, as well as the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system.
[0088] Further, refer to Figure 5 , step 31 may specifically include step 311 and step 312, wherein:
[0089] Step 311 projects the dedistorted image into the Gaussian coordinate system based on the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system to obtain the projection of the dedistorted image in the Gaussian coordinate system;
[0090] In step 312 , based on the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system, the projection of the dedistorted image in the Gaussian coordinate system is mapped to the road surface coordinate system to obtain an inverse perspective transformed image of the dedistorted image.
[0091] Specifically, there are currently four coordinate systems, namely the coordinate system of the roadside camera, the Gaussian coordinate system, the road surface coordinate system, and the pixel coordinate system of the image; among them, the pixel coordinate system takes the upper left corner of the image as the origin and the unit is pixel; the Gaussian coordinate system describes the position of the roadside camera; the road surface coordinate system, that is, the world coordinate system, describes the actual position relationship of the target, and its origin is a point in the point cloud data, which can be selected by the user or a preset point; the coordinate system of the roadside camera connects the physical coordinate system of the image and the road surface coordinate system, where the optical center of the lens is the origin, which is also the projection center.
[0092] In one embodiment of the present application, the transformation matrix specifically includes: Characterize the transformation relationship between the road surface coordinate system and the Gaussian coordinate system; Characterize the transformation relationship between the coordinate system of the roadside camera and the Gaussian coordinate system; The transformation relationship between the coordinate system of the roadside camera and the road surface coordinate system; Characterizes the correspondence between the coordinate system of the pixel in the dedistorted image and the road coordinate system. Where Pw is the three-dimensional coordinate point in the Gaussian coordinate system, Proad is the three-dimensional coordinate point in the road coordinate system, R1 is the rotation matrix between the road coordinate system and the Gaussian coordinate system, T1 is the translation between the road coordinate system and the Gaussian coordinate system, Pcamera is the coordinate point in the roadside camera's coordinate system, R2 is the rotation matrix between the roadside camera's coordinate system and the Gaussian coordinate system, T2 is the translation between the roadside camera's coordinate system and the Gaussian coordinate system, Kcamera is the intrinsic parameter matrix of the roadside camera, Puv is the pixel coordinate in the dedistorted image, Zc is the distance between the feature object and the camera's optical center, R1.inv is the inverse matrix between the road coordinate system and the Gaussian coordinate system, and R2.inv is the inverse matrix between the roadside camera's coordinate system and the Gaussian coordinate system.
[0093] Specifically, refer to Figure 6 Based on the transformation matrix, each pixel in the dedistorted image is mapped to the road coordinate system to complete the inverse perspective transformation. The pixel in the dedistorted image is mapped to the road coordinate system to obtain the inverse perspective transformed image (IPM image). The size and shape of the far and near targets in the perspective transformed image are consistent with the actual targets.
[0094] Further, refer to Figure 7 , Figure 7The training step process of the target segmentation model may specifically include: obtaining multiple training images, where the training images are images collected by multiple roadside cameras, and then performing dedistortion processing on each training image to obtain a dedistorted image of each training image; performing inverse perspective transformation processing on each dedistorted image based on regional point cloud data corresponding to each dedistorted image to obtain an inverse perspective transformed image of each dedistorted image; annotating the inverse perspective transformed image of each dedistorted image, and using each annotated inverse perspective transformed image as a training sample to train the initial model until the preset training conditions are met to obtain a trained target segmentation image.
[0095] Specifically, the process of processing each training image to obtain a training sample is to perform dedistortion processing and inverse perspective transformation processing on each training sample to obtain an inverse perspective transformed image. The inverse perspective transformed image corresponding to each training image is used as a training sample to train the initial network model, thereby obtaining a target segmentation model.
[0096] The steps of performing dedistortion processing and inverse perspective transformation processing on each training image are the same as the steps of performing dedistortion processing on the image to be identified in step 20 and performing inverse perspective transformation processing on the dedistorted image in step 30. Specifically, it may include performing dedistortion processing and inverse perspective transformation processing on each training image. The dedistortion processing includes internal parameter calibration, wherein the internal parameter is an optical parameter, and the optical parameter includes the principal point, focal length, and distortion coefficient when the roadside camera shoots the image to be identified.
[0097] In one embodiment of the present application, if the image attributes of the image to be identified taken by the roadside camera contain optical parameters, the image attributes of the image to be identified can be directly read to obtain the optical parameters.
[0098] In another embodiment of the present application, a black and white checkerboard object or image can be placed in the collection area of the roadside camera, and then the roadside camera can be controlled to collect images. At this time, the training image collected by the roadside camera includes a black and white checkerboard target, and then the image is imported into MATLAB (Matrix Laboratory) software to obtain the optical parameters of the roadside camera.
[0099] After determining the optical parameters, the training image and the optical parameters can be input into the OpenCV (Open Source Computer Vision Library) software respectively, and the cv2.undistort() function can be used to remove the distortion, thereby obtaining the dedistorted image of the training image.
[0100] In one embodiment, the dedistorted image corresponding to each training image is calibrated with external parameters to perform inverse perspective transformation processing, wherein the external parameter is a transformation matrix, and the transformation matrix includes the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system, and the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system.
[0101] Obtain the regional point cloud data corresponding to each training image. The target contained in the regional point cloud data should be larger than the target contained in the dedistorted image. Therefore, the target in the point cloud data cannot directly establish a corresponding relationship with the target in the dedistorted image. Specifically, several control points selected in the point cloud can be selected by the user; or the corner points of zebra crossings and arrows in the regional point cloud data can be automatically identified as control points based on the point cloud parsing software. Furthermore, the corner points of the dedistorted image are identified to determine the corner points of the road target, and then the corner points are calibrated and displayed. By establishing the corresponding relationship between the corner points and the control points, the transformation matrix can be determined.
[0102] Since the coordinate system of the roadside camera and the road surface coordinate system cannot be converted directly, in order to more accurately determine the conversion relationship between the two coordinate systems, a Gaussian coordinate system is introduced in the embodiment of the present application to perform relay conversion. Therefore, the transformation matrix specifically includes the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system, as well as the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system.
[0103] In one embodiment of the present application, the transformation matrix specifically includes: The transformation relationship between the road surface coordinate system and the Gaussian coordinate system; Characterize the transformation relationship between the coordinate system of the roadside camera and the Gaussian coordinate system; Characterize the transformation relationship between the coordinate system of the roadside camera and the road surface coordinate system; Characterizes the correspondence between the coordinate system of the pixel in the dedistorted image and the road coordinate system. Where Pw is the three-dimensional coordinate point in the Gaussian coordinate system, Proad is the three-dimensional coordinate point in the road coordinate system, R1 is the rotation matrix between the road coordinate system and the Gaussian coordinate system, T1 is the translation between the road coordinate system and the Gaussian coordinate system, Pcamera is the coordinate point in the roadside camera's coordinate system, R2 is the rotation matrix between the roadside camera's coordinate system and the Gaussian coordinate system, T2 is the translation between the roadside camera's coordinate system and the Gaussian coordinate system, Kcamera is the intrinsic parameter matrix of the roadside camera, Puv is the pixel coordinate in the dedistorted image, Zc is the distance between the feature object and the camera's optical center, R1.inv is the inverse matrix between the road coordinate system and the Gaussian coordinate system, and R2.inv is the inverse matrix between the roadside camera's coordinate system and the Gaussian coordinate system.
[0104] Specifically, based on the transformation matrix, each pixel in the dedistorted image is mapped to the road coordinate system to complete the inverse perspective transformation, and the pixel in the dedistorted image is mapped to the road coordinate system to obtain the inverse perspective transformed image (IPM image). The size and shape of the far and near targets in the perspective transformed image are consistent with the actual targets.
[0105] In an embodiment of the present application, the target segmentation model is trained based on the inverse perspective transformation image as a training sample. In specific applications, the inverse perspective transformation image is processed by the trained target segmentation model to facilitate the extraction of more accurate road targets.
[0106] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0107] Based on the same inventive concept, the embodiment of the present application also provides a target extraction device for implementing the target extraction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more target extraction device embodiments provided below can refer to the limitations on the target extraction method above, and will not be repeated here.
[0108] In an exemplary embodiment, Figure 8 As shown, a target extraction device is provided, including a to-be-recognized image acquisition module 801, a dedistortion module 802, an inverse perspective transformation module 803 and a segmentation module 804, wherein:
[0109] The to-be-recognized image acquisition module 801 is used to acquire the to-be-recognized image when generating a map update instruction, where the to-be-recognized image is an image captured by a roadside camera;
[0110] The dedistortion module 802 is used to perform dedistortion processing on the image to be identified to obtain a dedistorted image;
[0111] An inverse perspective transformation module 803 is used to perform inverse perspective transformation processing on the dedistorted image based on the regional point cloud data corresponding to the dedistorted image to obtain an inverse perspective transformed image of the dedistorted image;
[0112] The segmentation module 804 is used to input the inverse perspective transformed image into the trained target segmentation model to obtain various road targets.
[0113] In the embodiment of the present application, the device can perform dedistortion processing on the image to be identified, obtain a dedistorted image that can eliminate the change in the target shape, and then perform inverse perspective transformation processing on the dedistorted image to obtain an image from a bird's-eye view. From a bird's-eye view, each target is in the same plane. Therefore, the inverse perspective transformation image can eliminate the change in the shape, position, size and proportion of the remote target in the image to be identified taken by the roadside camera. Furthermore, by extracting the inverse perspective transformation image through the target segmentation model, the actual position, shape and size relationship of each road target in the image can be extracted more accurately.
[0114] In one embodiment, the de-distortion module 802 is specifically used for:
[0115] Determine the optical parameters of the image to be recognized, the optical parameters including the principal point, focal length and distortion coefficient;
[0116] The image to be identified is dedistorted based on the optical parameters to obtain a dedistorted image.
[0117] In one embodiment, the inverse perspective transformation module 803 is used to:
[0118] The dedistorted image is fused with the corresponding regional point cloud data to obtain a transformation matrix corresponding to the dedistorted image. The regional point cloud data is the point cloud data corresponding to the area captured by the image to be identified. The transformation matrix represents the rotation and translation relationship between the roadside camera coordinate system and the road surface coordinate system.
[0119] The dedistorted image is projected from the roadside camera coordinate system to the ground coordinate system based on the transformation matrix to obtain an inverse perspective transformed image of the dedistorted image.
[0120] In one embodiment, the transformation matrix includes the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system, and the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system. The inverse perspective transformation module 803 is further used to:
[0121] Based on the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system, the dedistorted image is projected into the Gaussian coordinate system to obtain the projection of the dedistorted image in the Gaussian coordinate system;
[0122] Based on the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system, the projection of the dedistorted image in the Gaussian coordinate system is mapped to the road surface coordinate system to obtain an inverse perspective transformation image of the dedistorted image.
[0123] In one embodiment, the dedistortion module 803 is further used to fuse the dedistorted image with the corresponding regional point cloud data to obtain a transformation matrix corresponding to the dedistorted image, including:
[0124] Determine a number of control points in the regional point cloud data;
[0125] Perform corner point recognition in the dedistorted image and determine several key points;
[0126] Based on the correspondence between the control points and the key points, the transformation matrix is determined.
[0127] In one embodiment, the target extraction device further includes a training module, specifically for
[0128] Acquire multiple training images, where the training images are images captured by multiple roadside cameras;
[0129] Performing dedistortion processing on each training image to obtain a dedistorted image of each training image;
[0130] Based on the regional point cloud data corresponding to the dedistorted image, the dedistorted image is subjected to inverse perspective transformation processing to obtain an inverse perspective transformed image of the dedistorted image;
[0131] The inverse perspective transformed image of each dedistorted image is annotated, and each annotated inverse perspective transformed image is used as a training sample to train the initial model until the preset training conditions are met to obtain a trained target segmentation image.
[0132] In one embodiment, the target extraction device further includes an instruction determination module, which is used to:
[0133] For each roadside camera, obtain a training image taken by the roadside camera;
[0134] Extract elements from the training images to obtain road surface elements;
[0135] Obtaining original road surface elements, where the original road surface elements are road surface elements obtained by extracting features from a training image last taken by a roadside camera;
[0136] Compare the road surface elements with the original road surface elements for similarity;
[0137] Based on the comparison result, it is determined whether to generate a map update instruction.
[0138] In one embodiment, the instruction determination module is specifically used to:
[0139] If the comparison result shows that the similarity is greater than or equal to the set threshold, no map update instruction is generated;
[0140] If the comparison result is that the similarity is less than the set threshold, the location information of the roadside camera is obtained, and a map update instruction is generated based on the location information. The map update instruction is used to instruct the point cloud collection vehicle to collect regional point cloud data based on the location information.
[0141] Each module in the above target extraction device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0142] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as training images and images to be identified. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a target extraction method is implemented.
[0143] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0144] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0145] When generating a map update instruction, an image to be identified is obtained, where the image to be identified is an image captured by a roadside camera;
[0146] Performing dedistortion processing on the image to be identified to obtain a dedistorted image;
[0147] Performing inverse perspective transformation processing on the dedistorted image based on regional point cloud data corresponding to the dedistorted image to obtain an inverse perspective transformed image of the dedistorted image;
[0148] The inverse perspective transformed image is input into the trained target segmentation model to obtain various road targets.
[0149] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0150] The image to be identified is subjected to dedistortion processing to obtain a dedistorted image, including:
[0151] Determine the optical parameters of the image to be recognized, the optical parameters including the principal point, focal length and distortion coefficient;
[0152] The image to be identified is dedistorted based on the optical parameters to obtain a dedistorted image.
[0153] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0154] The dedistorted image is subjected to inverse perspective transformation processing based on the regional point cloud data corresponding to the dedistorted image to obtain an inverse perspective transformation image of the dedistorted image, including:
[0155] The dedistorted image is fused with the corresponding regional point cloud data to obtain a transformation matrix corresponding to the dedistorted image. The regional point cloud data is the point cloud data corresponding to the area captured by the image to be identified. The transformation matrix represents the rotation and translation relationship between the roadside camera coordinate system and the road surface coordinate system.
[0156] The dedistorted image is projected from the roadside camera coordinate system to the ground coordinate system based on the transformation matrix to obtain an inverse perspective transformed image of the dedistorted image.
[0157] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0158] The transformation matrix includes the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system, and the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system. Based on the transformation matrix, the dedistorted image is projected from the roadside camera coordinate system to the ground coordinate system to obtain the inverse perspective transformation image of the dedistorted image, including:
[0159] Based on the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system, the dedistorted image is projected into the Gaussian coordinate system to obtain the projection of the dedistorted image in the Gaussian coordinate system;
[0160] Based on the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system, the projection of the dedistorted image in the Gaussian coordinate system is mapped to the road surface coordinate system to obtain an inverse perspective transformation image of the dedistorted image.
[0161] In one embodiment, when the processor executes the computer program, the following steps are also implemented:
[0162] The dedistorted image is fused with the corresponding regional point cloud data to obtain the transformation matrix corresponding to the dedistorted image, including:
[0163] Determine a number of control points in the regional point cloud data;
[0164] Perform corner point recognition in the dedistorted image and determine several key points;
[0165] Based on the correspondence between the control points and the key points, the transformation matrix is determined.
[0166] In one embodiment, when the processor executes the computer program, the following steps are also implemented:
[0167] The training steps of the target segmentation model also include:
[0168] Acquire multiple training images, where the training images are images captured by multiple roadside cameras;
[0169] Performing dedistortion processing on each training image to obtain a dedistorted image of each training image;
[0170] Based on the regional point cloud data corresponding to the dedistorted image, the dedistorted image is subjected to inverse perspective transformation processing to obtain an inverse perspective transformed image of the dedistorted image;
[0171] The inverse perspective transformed image of each dedistorted image is annotated, and each annotated inverse perspective transformed image is used as a training sample to train the initial model until the preset training conditions are met to obtain a trained target segmentation image.
[0172] In one embodiment, when the processor executes the computer program, the following steps are also implemented:
[0173] Before obtaining the image to be recognized, it also includes:
[0174] For each roadside camera, obtain a training image taken by the roadside camera;
[0175] Extract elements from the training images to obtain road surface elements;
[0176] Obtaining original road surface elements, where the original road surface elements are road surface elements obtained by extracting features from a training image last taken by a roadside camera;
[0177] Compare the road surface elements with the original road surface elements for similarity;
[0178] Based on the comparison result, it is determined whether to generate a map update instruction.
[0179] In one embodiment, when the processor executes the computer program, the following steps are also implemented:
[0180] Determining whether to generate a map update instruction based on the comparison result includes:
[0181] If the comparison result shows that the similarity is greater than or equal to the set threshold, no map update instruction is generated;
[0182] If the comparison result is that the similarity is less than the set threshold, the location information of the roadside camera is obtained, and a map update instruction is generated based on the location information.
[0183] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0184] When generating a map update instruction, an image to be identified is obtained, where the image to be identified is an image captured by a roadside camera;
[0185] Performing dedistortion processing on the image to be identified to obtain a dedistorted image;
[0186] Performing inverse perspective transformation processing on the dedistorted image based on regional point cloud data corresponding to the dedistorted image to obtain an inverse perspective transformed image of the dedistorted image;
[0187] The inverse perspective transformed image is input into the trained target segmentation model to obtain various road targets.
[0188] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0189] The image to be identified is subjected to dedistortion processing to obtain a dedistorted image, including:
[0190] Determine the optical parameters of the image to be recognized, the optical parameters including the principal point, focal length and distortion coefficient;
[0191] The image to be identified is dedistorted based on the optical parameters to obtain a dedistorted image.
[0192] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0193] The dedistorted image is subjected to inverse perspective transformation processing based on the regional point cloud data corresponding to the dedistorted image to obtain an inverse perspective transformation image of the dedistorted image, including:
[0194] The dedistorted image is fused with the corresponding regional point cloud data to obtain a transformation matrix corresponding to the dedistorted image. The regional point cloud data is the point cloud data corresponding to the area captured by the image to be identified. The transformation matrix represents the rotation and translation relationship between the roadside camera coordinate system and the road surface coordinate system.
[0195] The dedistorted image is projected from the roadside camera coordinate system to the ground coordinate system based on the transformation matrix to obtain an inverse perspective transformed image of the dedistorted image.
[0196] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: the transformation matrix includes the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system, and the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system; based on the transformation matrix, the dedistorted image is projected from the roadside camera coordinate system to the ground coordinate system to obtain an inverse perspective transformation image of the dedistorted image, including:
[0197] Based on the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system, the dedistorted image is projected into the Gaussian coordinate system to obtain the projection of the dedistorted image in the Gaussian coordinate system;
[0198] Based on the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system, the projection of the dedistorted image in the Gaussian coordinate system is mapped to the road surface coordinate system to obtain an inverse perspective transformation image of the dedistorted image.
[0199] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: fusing the dedistorted image with the corresponding regional point cloud data to obtain a transformation matrix corresponding to the dedistorted image, including:
[0200] Determine a number of control points in the regional point cloud data;
[0201] Perform corner recognition in the dedistorted image and determine several key points;
[0202] Based on the correspondence between the control points and the key points, the transformation matrix is determined.
[0203] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: a target segmentation model training step, further comprising:
[0204] Acquire multiple training images, where the training images are images captured by multiple roadside cameras;
[0205] Performing dedistortion processing on each training image to obtain a dedistorted image of each training image;
[0206] Based on the regional point cloud data corresponding to the dedistorted image, the dedistorted image is subjected to inverse perspective transformation processing to obtain an inverse perspective transformed image of the dedistorted image;
[0207] The inverse perspective transformed image of each dedistorted image is annotated, and each annotated inverse perspective transformed image is used as a training sample to train the initial model until the preset training conditions are met to obtain a trained target segmentation image.
[0208] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: before acquiring the image to be identified, the following steps are further included:
[0209] For each roadside camera, obtain a training image taken by the roadside camera;
[0210] Extract elements from the training images to obtain road surface elements;
[0211] Obtaining original road surface elements, where the original road surface elements are road surface elements obtained by extracting features from a training image last taken by a roadside camera;
[0212] Compare the road surface elements with the original road surface elements for similarity;
[0213] Based on the comparison result, it is determined whether to generate a map update instruction.
[0214] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining whether to generate a map update instruction based on the comparison result, including:
[0215] If the comparison result shows that the similarity is greater than or equal to the set threshold, no map update instruction is generated;
[0216] If the comparison result is that the similarity is less than the set threshold, the location information of the roadside camera is obtained, and a map update instruction is generated based on the location information.
[0217] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0218] When generating a map update instruction, an image to be identified is obtained, where the image to be identified is an image captured by a roadside camera;
[0219] Performing dedistortion processing on the image to be identified to obtain a dedistorted image;
[0220] Performing inverse perspective transformation processing on the dedistorted image based on regional point cloud data corresponding to the dedistorted image to obtain an inverse perspective transformed image of the dedistorted image;
[0221] The inverse perspective transformed image is input into the trained target segmentation model to obtain various road targets.
[0222] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0223] The image to be identified is subjected to dedistortion processing to obtain a dedistorted image, including:
[0224] Determine the optical parameters of the image to be recognized, the optical parameters including the principal point, focal length and distortion coefficient;
[0225] The image to be identified is dedistorted based on the optical parameters to obtain a dedistorted image.
[0226] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0227] The dedistorted image is subjected to inverse perspective transformation processing based on the regional point cloud data corresponding to the dedistorted image to obtain an inverse perspective transformation image of the dedistorted image, including:
[0228] The dedistorted image is fused with the corresponding regional point cloud data to obtain a transformation matrix corresponding to the dedistorted image. The regional point cloud data is the point cloud data corresponding to the area captured by the image to be identified. The transformation matrix represents the rotation and translation relationship between the roadside camera coordinate system and the road surface coordinate system.
[0229] The dedistorted image is projected from the roadside camera coordinate system to the ground coordinate system based on the transformation matrix to obtain an inverse perspective transformed image of the dedistorted image.
[0230] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: the transformation matrix includes the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system, and the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system; based on the transformation matrix, the dedistorted image is projected from the roadside camera coordinate system to the ground coordinate system to obtain an inverse perspective transformation image of the dedistorted image, including:
[0231] Based on the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system, the dedistorted image is projected into the Gaussian coordinate system to obtain the projection of the dedistorted image in the Gaussian coordinate system;
[0232] Based on the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system, the projection of the dedistorted image in the Gaussian coordinate system is mapped to the road surface coordinate system to obtain an inverse perspective transformation image of the dedistorted image.
[0233] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: fusing the dedistorted image with the corresponding regional point cloud data to obtain a transformation matrix corresponding to the dedistorted image, including:
[0234] Determine a number of control points in the regional point cloud data;
[0235] Perform corner point recognition in the dedistorted image and determine several key points;
[0236] Based on the correspondence between the control points and the key points, the transformation matrix is determined.
[0237] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: a target segmentation model training step, further comprising:
[0238] Acquire multiple training images, where the training images are images captured by multiple roadside cameras;
[0239] Performing dedistortion processing on each training image to obtain a dedistorted image of each training image;
[0240] Based on the regional point cloud data corresponding to the dedistorted image, the dedistorted image is subjected to inverse perspective transformation processing to obtain an inverse perspective transformed image of the dedistorted image;
[0241] The inverse perspective transformed image of each dedistorted image is annotated, and each annotated inverse perspective transformed image is used as a training sample to train the initial model until the preset training conditions are met to obtain a trained target segmentation image.
[0242] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: before acquiring the image to be identified, the following steps are further included:
[0243] For each roadside camera, obtain a training image taken by the roadside camera;
[0244] Extract elements from the training images to obtain road surface elements;
[0245] Obtaining original road surface elements, where the original road surface elements are road surface elements obtained by extracting features from a training image last taken by a roadside camera;
[0246] Compare the road surface elements with the original road surface elements for similarity;
[0247] Based on the comparison result, it is determined whether to generate a map update instruction.
[0248] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining whether to generate a map update instruction based on the comparison result, including:
[0249] If the comparison result shows that the similarity is greater than or equal to the set threshold, no map update instruction is generated;
[0250] If the comparison result is that the similarity is less than the set threshold, the location information of the roadside camera is obtained, and a map update instruction is generated based on the location information.
[0251] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0252] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0253] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0254] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A target extraction method, characterized in that: The method comprises: When generating a map update instruction, obtaining an image to be identified, where the image to be identified is an image captured by a roadside camera; Performing dedistortion processing on the image to be identified to obtain a dedistorted image; Performing inverse perspective transformation processing on the dedistorted image based on regional point cloud data corresponding to the dedistorted image to obtain an inverse perspective transformed image of the dedistorted image; The inverse perspective transformed image is input into a trained target segmentation model to obtain various road targets.
2. The method according to claim 1, characterized in that The step of performing a dedistortion process on the image to be identified to obtain a dedistorted image includes: Determining optical parameters of the image to be recognized, wherein the optical parameters include a principal point, a focal length, and a distortion coefficient; Dedistortion processing is performed on the image to be recognized based on the optical parameters to obtain a dedistorted image.
3. The method according to claim 2, characterized in that The performing inverse perspective transformation processing on the dedistorted image based on the regional point cloud data corresponding to the dedistorted image to obtain an inverse perspective transformed image of the dedistorted image includes: The dedistorted image is fused with the corresponding regional point cloud data to obtain a transformation matrix corresponding to the dedistorted image, wherein the regional point cloud data is the point cloud data corresponding to the region captured by the image to be identified, and the transformation matrix represents the rotation and translation relationship between the roadside camera coordinate system and the road surface coordinate system; The dedistorted image is projected from the roadside camera coordinate system to a ground coordinate system based on the transformation matrix to obtain an inverse perspective transformed image of the dedistorted image.
4. The method according to claim 3, characterized in that The transformation matrix includes a rotation and translation relationship between a roadside camera coordinate system and a Gaussian coordinate system, and a rotation and translation relationship between a Gaussian coordinate system and a road surface coordinate system. Based on the transformation matrix, the dedistorted image is projected from the roadside camera coordinate system to a ground coordinate system to obtain an inverse perspective transformed image of the dedistorted image, including: Based on the rotation and translation relationship between the roadside camera coordinate system and the Gaussian coordinate system, projecting the dedistorted image into the Gaussian coordinate system to obtain a projection of the dedistorted image in the Gaussian coordinate system; Based on the rotation and translation relationship between the Gaussian coordinate system and the road surface coordinate system, the projection of the dedistorted image in the Gaussian coordinate system is mapped to the road surface coordinate system to obtain an inverse perspective transformation image of the dedistorted image.
5. The method according to any one of claims 1 to 4, characterized in that: The step of fusing the dedistorted image with the corresponding regional point cloud data to obtain a transformation matrix corresponding to the dedistorted image includes: Determining a number of control points in the regional point cloud data; Performing corner point recognition in the dedistorted image to determine a number of key points; A transformation matrix is determined based on the correspondence between the control points and the key points.
6. The method according to claim 1, characterized in that The training step of the target segmentation model also includes: Acquire multiple training images, where the training images are images captured by multiple roadside cameras; Performing dedistortion processing on each of the training images to obtain a dedistorted image of each of the training images; Based on the regional point cloud data corresponding to the dedistorted image, the dedistorted image is subjected to inverse perspective transformation processing to obtain an inverse perspective transformed image of the dedistorted image; The inverse perspective transformed image of each dedistorted image is annotated, and each annotated inverse perspective transformed image is used as a training sample to train the initial model until the preset training conditions are met to obtain a trained target segmentation image.
7. The method according to claim 1, characterized in that Before acquiring the image to be recognized, the method further includes: For each roadside camera, obtaining a training image taken by the roadside camera; Extracting elements from the training image to obtain road surface elements; Acquire original road surface elements, where the original road surface elements are road surface elements obtained by extracting features from a training image last taken by the roadside camera; Comparing the road surface elements with the original road surface elements for similarity; Based on the comparison result, it is determined whether to generate a map update instruction.
8. The method according to claim 7, characterized in that The determining whether to generate a map update instruction based on the comparison result includes: If the comparison result shows that the similarity is greater than or equal to the set threshold, no map update instruction is generated; If the comparison result is that the similarity is less than the set threshold, the position information of the roadside camera is obtained, and a map update instruction is generated based on the position information, and the map update instruction is used to instruct the point cloud collection vehicle to collect regional point cloud data based on the position information.
9. A remote target extraction device, characterized in that: The device comprises: An image acquisition module to be identified, used to acquire an image to be identified when generating a map update instruction, wherein the image to be identified is an image captured by a roadside camera; A dedistortion module, used for performing dedistortion processing on the image to be identified to obtain a dedistorted image; An inverse perspective transformation module, configured to perform inverse perspective transformation processing on the dedistorted image based on regional point cloud data corresponding to the dedistorted image, so as to obtain an inverse perspective transformed image of the dedistorted image; The segmentation module is used to input the inverse perspective transformed image into a trained target segmentation model to obtain various road targets.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
