Target event positioning method and device, computer device and storage medium
By acquiring the camera's intrinsic and extrinsic parameters and constructing a transformation matrix, combined with the camera's actual position, the problem of being unable to locate target events in existing technologies is solved, enabling precise location of target events and traffic flow management.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUOQIZHITU (BEIJING) TECH CO LTD
- Filing Date
- 2022-12-23
- Publication Date
- 2026-06-02
AI Technical Summary
Existing target event recognition methods can only identify whether a target event has occurred in road image data, but cannot locate the actual location of the target event, which makes it impossible for cloud servers to effectively avoid traffic congestion.
By acquiring the camera's intrinsic and extrinsic parameters, a transformation matrix is constructed. Combined with the camera's actual position, multiple coordinate transformations are performed to locate the target event.
It achieves precise location of target events and can upload their actual location to the cloud server, effectively avoiding traffic congestion.
Smart Images
Figure CN115937315B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of vehicle networking technology, and in particular to a target event localization method, apparatus, computer equipment, storage medium, and computer program product. Background Technology
[0002] Traffic congestion is easily caused by events such as traffic accidents or road construction. It is necessary to promptly locate these events and obtain their actual positions on the affected road segment. These positions are then uploaded to a cloud server, which notifies the public of the occurrence of the event and its location to prevent congestion in that area.
[0003] Current target event recognition methods involve acquiring road image data captured by surveillance cameras, identifying whether a target event has occurred in the road image data, and uploading the target event message to a cloud server if a target event is present in the road image data.
[0004] However, current target event recognition methods can only identify road image data and determine whether a target event has occurred in the road image data, but cannot locate the target event. Therefore, cloud servers cannot announce the actual location of the target event to avoid traffic congestion on that road segment. Summary of the Invention
[0005] Therefore, it is necessary to provide a target event localization method, apparatus, computer device, computer-readable storage medium, and computer program product to address the aforementioned technical problems.
[0006] Firstly, this application provides a method for locating a target event. The method includes:
[0007] The system acquires real-time image data and intrinsic parameters of the camera, and determines extrinsic parameters of the camera based on the image data; the image data represents road information of the monitored target road.
[0008] Image detection is performed on the image data to obtain image detection results. When the image detection results indicate that a target event has occurred on the monitored target road, the first position coordinates of the pixel representing the target event in the image coordinate system are output.
[0009] Based on the intrinsic and extrinsic parameters, a transformation matrix for the camera is constructed; based on the transformation matrix, the first position coordinates of the target event in the image coordinate system are converted into second position coordinates in the camera coordinate system.
[0010] Obtain the actual position of the camera, and based on the actual position of the camera, perform coordinate transformation on the second position coordinates of the target event in the camera coordinate system to obtain the actual position coordinates of the target event.
[0011] In one embodiment, the camera's extrinsic parameters include the camera's pitch angle, yaw angle, roll angle, and mounting height; determining the camera's extrinsic parameters based on the image data includes:
[0012] According to the supervised image semantic segmentation algorithm, the lane lines in the image data are segmented to obtain the segmented lane line observation values. The disappearance point is determined by the intersection of the lane line observation values, and the position of the disappearance point in the image coordinate system is obtained.
[0013] Based on the projection model, matrix operations are performed on the position of the disappearance point in the image coordinate system and the intrinsic parameters to obtain the pitch angle and yaw angle of the camera.
[0014] The actual length of a single lane is obtained, and the pixel length of the single lane in the image data and the actual length are calculated using the least squares method to obtain the roll angle and installation height of the camera.
[0015] In one embodiment, before performing image detection on the image data to obtain an image detection result, and outputting the first position coordinates of the pixel representing the target event in the image coordinate system when the image detection result indicates that a target event has occurred on the monitored target road, the process includes:
[0016] Obtain the image sample set corresponding to the image data, wherein each image sample in the image sample set contains information indicating the position of the pixel representing the target event in the image coordinate system;
[0017] The preset deep learning model is trained based on the image sample set to obtain the trained result data.
[0018] If the precision and recall rates contained in the result data meet a preset threshold, a trained deep learning model is obtained.
[0019] In one embodiment, the step of performing image detection on the image data to obtain an image detection result, and when the image detection result indicates that a target event has occurred on the monitored target road, outputting the first position coordinates of the pixel representing the target event in the image coordinate system, includes:
[0020] The trained deep learning model is used to perform image detection on the image data to obtain image detection results.
[0021] When the image detection result indicates that a target event has occurred on the monitored target road, the first position coordinates of the pixel of the target event in the image coordinate system are output.
[0022] In one embodiment, obtaining the actual location where the camera is installed includes:
[0023] Multiple control points are selected from the image data to obtain the positions of the multiple control points in the image coordinate system;
[0024] Obtain point cloud data of the target road, and determine the actual positions of the plurality of control points in the point cloud data;
[0025] Based on a preset error function, the actual positions of the multiple control points and their positions in the image coordinate system are processed to obtain the actual position where the camera is installed.
[0026] In one embodiment, obtaining the actual position of the camera and performing a coordinate transformation on the second position of the target event in the camera coordinate system based on the actual position of the camera to obtain the actual position of the target event includes:
[0027] Based on the extrinsic parameters of the camera, construct the rotation matrix of the camera;
[0028] A matrix multiplication operation is performed on the rotation matrix of the camera and the second position of the target event in the camera coordinate system. The result of the matrix multiplication operation is then added to the actual position of the camera to obtain the actual position of the target event.
[0029] Secondly, this application also provides a target event localization device. The device includes:
[0030] The acquisition module is used to acquire image data and intrinsic parameters of the camera in real time, and determine the extrinsic parameters of the camera based on the image data; the image data represents the road information of the monitored target road.
[0031] The detection module is used to perform image detection on the image data to obtain image detection results. When the image detection results indicate that a target event has occurred on the monitored target road, the module outputs the first position coordinates of the pixel representing the target event in the image coordinate system.
[0032] A construction module is used to construct the transformation matrix of the camera based on the intrinsic parameters and the extrinsic parameters; and to convert the first position coordinates of the target event in the image coordinate system into the second position coordinates in the camera coordinate system based on the transformation matrix.
[0033] The transformation module is used to obtain the actual position of the camera, and perform coordinate transformation on the second position coordinates of the target event in the camera coordinate system based on the actual position of the camera to obtain the actual position coordinates of the target event.
[0034] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:
[0035] The system acquires real-time image data and intrinsic parameters of the camera, and determines extrinsic parameters of the camera based on the image data; the image data represents road information of the monitored target road.
[0036] Image detection is performed on the image data to obtain image detection results. When the image detection results indicate that a target event has occurred on the monitored target road, the first position coordinates of the pixel representing the target event in the image coordinate system are output.
[0037] Based on the intrinsic and extrinsic parameters, a transformation matrix for the camera is constructed; based on the transformation matrix, the first position coordinates of the target event in the image coordinate system are converted into second position coordinates in the camera coordinate system.
[0038] Obtain the actual position of the camera, and based on the actual position of the camera, perform coordinate transformation on the second position coordinates of the target event in the camera coordinate system to obtain the actual position coordinates of the target event.
[0039] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:
[0040] The system acquires real-time image data and intrinsic parameters of the camera, and determines extrinsic parameters of the camera based on the image data; the image data represents road information of the monitored target road.
[0041] Image detection is performed on the image data to obtain image detection results. When the image detection results indicate that a target event has occurred on the monitored target road, the first position coordinates of the pixel representing the target event in the image coordinate system are output.
[0042] Based on the intrinsic and extrinsic parameters, a transformation matrix for the camera is constructed; based on the transformation matrix, the first position coordinates of the target event in the image coordinate system are converted into second position coordinates in the camera coordinate system.
[0043] Obtain the actual position of the camera, and based on the actual position of the camera, perform coordinate transformation on the second position coordinates of the target event in the camera coordinate system to obtain the actual position coordinates of the target event.
[0044] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:
[0045] The system acquires real-time image data and intrinsic parameters of the camera, and determines extrinsic parameters of the camera based on the image data; the image data represents road information of the monitored target road.
[0046] Image detection is performed on the image data to obtain image detection results. When the image detection results indicate that a target event has occurred on the monitored target road, the first position coordinates of the pixel representing the target event in the image coordinate system are output.
[0047] Based on the intrinsic and extrinsic parameters, a transformation matrix for the camera is constructed; based on the transformation matrix, the first position coordinates of the target event in the image coordinate system are converted into second position coordinates in the camera coordinate system.
[0048] Obtain the actual position of the camera, and based on the actual position of the camera, perform coordinate transformation on the second position coordinates of the target event in the camera coordinate system to obtain the actual position coordinates of the target event.
[0049] The aforementioned target event localization method, apparatus, computer equipment, storage medium, and computer program product acquire image data and intrinsic parameters of the camera in real time, and determine the extrinsic parameters of the camera based on the image data. The image data represents road information of the monitored target road. Image detection is performed on the image data to obtain image detection results. When the image detection results indicate that a target event has occurred on the monitored target road, the first position coordinates of the pixel representing the target event in the image coordinate system are output. A transformation matrix of the camera is constructed based on the intrinsic and extrinsic parameters. Based on the transformation matrix, the first position coordinates of the target event in the image coordinate system are converted into a second position coordinate in the camera coordinate system. The actual position of the camera is obtained, and based on the actual position of the camera, the second position coordinates of the target event in the camera coordinate system are transformed to obtain the actual position coordinates of the target event. Using this method, a transformation matrix is constructed based on the camera's extrinsic and intrinsic parameters. Through the transformation matrix and the actual position of the camera, multiple coordinate transformations are performed on the first position coordinates of the target event in the image coordinate system to obtain the actual position of the target event, thus achieving target event localization. Attached Figure Description
[0050] Figure 1 This is a flowchart illustrating a target event localization method in one embodiment;
[0051] Figure 2 This is a schematic diagram of a chessboard calibration plate in one embodiment;
[0052] Figure 3 This is a schematic diagram illustrating the principle of inverse perspective transformation in one embodiment;
[0053] Figure 4 This is a flowchart illustrating the steps for determining extrinsic parameters in one embodiment;
[0054] Figure 5 This is a schematic diagram illustrating the relationship between pitch angle, yaw angle, and roll angle in one embodiment;
[0055] Figure 6 This is a flowchart illustrating the steps involved in training a deep learning model in one embodiment.
[0056] Figure 7 This is a flowchart illustrating the steps of image detection in one embodiment;
[0057] Figure 8 This is a flowchart illustrating the steps for obtaining the actual position of the camera in one embodiment;
[0058] Figure 9 This is a schematic diagram of selecting control points in one embodiment;
[0059] Figure 10 This is a flowchart illustrating the steps for obtaining the actual location of a target event in one embodiment;
[0060] Figure 11 This is a structural block diagram of a target event localization device in one embodiment;
[0061] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0063] In one embodiment, such as Figure 1 As shown, a target event localization method is provided. Taking the application of this method to a computer device as an example, this application embodiment does not limit the execution device for executing the target event localization method, and includes the following steps:
[0064] Step 102: Obtain the image data and intrinsic parameters of the camera acquired in real time, and determine the extrinsic parameters of the camera based on the image data.
[0065] Among them, image data represents the road information of the monitored target road.
[0066] In implementation, the computer equipment acquires real-time image data of the target road obtained by the camera, along with the camera's intrinsic parameters. Then, the computer equipment segments the lane lines in the image data and determines the positions of the disappearing points in the image coordinate system. Next, the computer equipment performs matrix operations on the positions of the disappearing points in the image coordinate system and the intrinsic parameters to obtain the camera's pitch and yaw angles. The computer equipment obtains the actual length of a single lane and calculates the actual length and pixel length of the single lane to obtain the camera's roll angle and mounting height. The computer equipment uses the camera's pitch, yaw, roll, and mounting height as the camera's extrinsic parameters.
[0067] Optionally, the computer device acquires the checkerboard calibration image data captured by the camera, such as... Figure 2 As shown. Then, the computer equipment processes the chessboard calibration board image data according to the preset Zhang Zhengyou calibration algorithm to obtain the camera's intrinsic parameters. The computer equipment stores the camera's intrinsic parameters in a preset storage location.
[0068] Step 104: Perform image detection on the image data to obtain the image detection result. When the image detection result indicates that a target event has occurred on the monitored target road, output the first position coordinates of the pixel representing the target event in the image coordinate system.
[0069] In implementation, the computer equipment stores deep learning algorithms. The computer equipment performs image detection on the image data according to the preset deep learning algorithm, obtaining the corresponding image detection results. When the image detection results indicate that a target event has occurred on the monitored target road, the computer equipment outputs the first position coordinates of the target event's pixel in the image coordinate system.
[0070] Optionally, the target event is an event that causes traffic congestion on the target road, such as a car accident or road construction. This embodiment of the application does not limit the specific content of the target event.
[0071] Step 106: Construct the camera transformation matrix based on the intrinsic and extrinsic parameters; based on the transformation matrix, convert the first position coordinates of the target event in the image coordinate system to the second position coordinates in the camera coordinate system.
[0072] In practice, the computer device constructs the camera's transformation matrix based on the camera's intrinsic and extrinsic parameters. Then, based on the camera's transformation matrix, the computer device performs coordinate transformation on the first position coordinates of the target event in the image coordinate system to obtain the second position coordinates of the target event in the camera coordinate system.
[0073] Specifically, such as Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the principle of inverse perspective transformation. Figure 3 In this context, Fw(Xw,Yw,Zw) represents a custom ground coordinate system, and Fc(Xc,Yc,Zc) represents the camera coordinate system. cu,cv are the camera optical centers in the camera's intrinsic parameters. h is the camera's installation height. α represents the pitch angle. β represents the yaw angle. The computer equipment constructs the transformation torque based on the camera's intrinsic and extrinsic parameters, as shown in formula (1).
[0074]
[0075] In formula (1), h is the camera mounting height. c1 = cosα, s1 = sinα, α is the pitch angle. c2 = cosβ, s2 = sinβ, β is the yaw angle. f u ,f v ... wiThe transformation torque is then calculated. The computer device then performs a matrix operation on the transformation torque and the first position coordinates of the target event in the image coordinate system, as shown in formula (2), to obtain the second position coordinates of the target event in the camera coordinate system.
[0076] p w =T wi p i (2)
[0077] In the above formula (2), T wi To change the torque. w p represents the second position coordinate of the target event in the camera coordinate system. i The coordinates of the first position of the target event in the image coordinate system.
[0078] Step 108: Obtain the actual position of the installed camera. Based on the actual position of the camera, perform coordinate transformation on the second position coordinates of the target event in the camera coordinate system to obtain the actual position coordinates of the target event.
[0079] In practice, the computer equipment acquires the actual position of the installed camera in the world coordinate system. Based on the actual position of the camera in the world coordinate system, the computer equipment performs a coordinate transformation on the second position coordinates of the target event in the camera coordinate system to obtain the actual position coordinates of the target event.
[0080] In the above target event localization method, a transformation matrix is constructed based on the camera's extrinsic and intrinsic parameters. By using the transformation matrix and the camera's actual position, multiple coordinate transformations are performed on the first position coordinates of the target event in the image coordinate system to obtain the actual position of the target event, thus achieving the localization of the target event.
[0081] In one embodiment, the camera's extrinsic parameters include the camera's pitch angle, yaw angle, roll angle, and mounting height. For example... Figure 4 As shown, the specific processing steps for determining the camera's extrinsic parameters based on image data in step 102 include:
[0082] Step 402: According to the supervised image semantic segmentation algorithm, the lane lines in the image data are segmented to obtain the segmented lane line observations. The disappearance point is determined by the intersection of the lane line observations, and the position of the disappearance point in the image coordinate system is obtained.
[0083] The labeled image data in the annotation set contains information representing the location of the labeled lane lines. The accuracy threshold is 90%.
[0084] In implementation, the computer device stores a supervised image semantic segmentation algorithm. The computer device segments the lane lines in the image data according to the supervised image semantic segmentation algorithm to obtain the observation values of two lane lines. Then, the computer device takes the position of the intersection point of the two lane line observation values in the image coordinate system as the position of the disappearance point in the image coordinate system, as shown in the following formula (3).
[0085] p∞=(u,v,1) T (3)
[0086] In the above formula (3), p∞ is the position of the disappearing point in the image coordinate system, u is the position of the disappearing point on the X-axis in the image coordinate system, v is the position of the disappearing point on the Y-axis in the image coordinate system, and l is the position of the disappearing point on the Z-axis in the image coordinate system.
[0087] Specifically, the computer device stores a supervised image semantic segmentation algorithm and an accuracy threshold. The computer device acquires a label set corresponding to the image data. Then, the computer device trains the supervised image semantic segmentation algorithm using the label set. When the accuracy reaches the accuracy threshold in the trained result data, the computer device obtains the trained supervised image semantic segmentation algorithm. The computer device segments the lane lines in the image data using the trained supervised image semantic segmentation algorithm, obtaining observations of two lane lines. Then, the computer device uses the position of the intersection point of the two lane line observations in the image coordinate system as the position of the disappearance point in the image coordinate system.
[0088] Optionally, the number of samples in the annotation set is 100,000, and the number of training rounds is 200. This embodiment of the application does not limit the number of samples in the annotation set or the number of training rounds.
[0089] Optionally, the accuracy threshold is determined by the training requirements, and the specific value of the accuracy threshold is not limited in this embodiment of the application.
[0090] Step 404: Based on the projection model, perform matrix operations on the position and intrinsic parameters of the disappearance point in the image coordinate system to obtain the camera's pitch and yaw angles.
[0091] The relationship between pitch angle, yaw angle, and roll angle is as follows: Figure 5 As shown. In Figure 5 In this context, VP is the vanishing point, the coordinate system with C as the origin is the camera coordinate system, and the coordinate system with W as the origin is the world coordinate system. Pitch represents the pitch angle, and yaw represents the yaw angle. Road Surface is the road boundary, and Lane Boundary is the lane line.
[0092] In implementation, the computer equipment processes the projection equations in the projection model to obtain the position of the vanishing point in the image coordinate system, the correspondence between the intrinsic parameter matrix and the pitch and yaw angles. Based on the position of the vanishing point in the image coordinate system, the computer equipment determines the pitch and yaw angles.
[0093] Specifically, the computer device constructs the camera's intrinsic parameter matrix based on the camera's intrinsic parameters. The computer device performs matrix operations on the projection model's functions to obtain the calculation formula for the third column of the rotation matrix between the camera and the ground, as shown in formula (4) below.
[0094]
[0095] In the above formula (4), r3 is the third column matrix in the rotation matrix between the camera and the ground, K-1 is the inverse matrix of the camera's intrinsic parameter matrix, and p∞ is the position of the disappearance point in the image coordinate system. Then, the computer device constructs the rotation matrix between the camera and the ground, as shown in formula (5).
[0096]
[0097] In the above formula (5), α represents the pitch angle, β represents the yaw angle, and γ represents the roll angle. The computer device obtains the analytical expressions for the pitch and yaw angles based on formulas (4) and (5), as shown in the following formula set (6). The computer device calculates the analytical expressions for the pitch and yaw angles to obtain the camera's pitch and yaw angles.
[0098]
[0099] In the above formula group (6), α represents the pitch angle, β represents the yaw angle, and R xz R in the above formula (4) xz R yz R in the above formula (4) yz R zz R in the above formula (4) zz .
[0100] Step 406: Obtain the actual length of the single lane, and calculate the pixel length and actual length of the single lane in the image data using the least squares method to obtain the camera's roll angle and installation height.
[0101] In practice, the computer equipment acquires the actual length of a single lane. The computer equipment stores the least squares method. Based on the least squares method, the computer equipment calculates the actual length of the single lane and the pixel length of the single lane in the image data to obtain the camera's roll angle and installation height.
[0102] In this embodiment, lane lines in the image data are segmented to determine disappearance points. Based on the projection model, the positions and intrinsic parameter matrices of these disappearance points in the image coordinate system are processed to obtain the camera's pitch and yaw angles. The camera's mounting height and roll angle are determined based on the actual length and pixel length of a single lane, yielding the camera's extrinsic parameters, which facilitate subsequent determination of the actual location of the target event.
[0103] In one embodiment, before performing image detection on image data, a deep learning model for image detection needs to be trained. Once the training results reach a preset threshold, the trained deep learning model is obtained, such as... Figure 6 As shown, before step 104 is executed, the specific processing steps of this target event localization method include:
[0104] Step 602: Obtain the image sample set corresponding to the image data.
[0105] Each image sample in the image sample set contains information about the position of the pixels representing the target event in the image coordinate system.
[0106] In practice, computer equipment acquires the image sample set corresponding to the image data.
[0107] Optionally, the pixels of the labeled target events in the image sample set can be labeled by machine or manually. This embodiment of the application does not limit the labeling method of the image sample set.
[0108] Optionally, the number of samples in the image sample set is 80,000. This embodiment of the application does not limit the number of samples in the image sample set.
[0109] Step 604: Train the preset deep learning model based on the image sample set to obtain the trained result data.
[0110] The deep learning model is YOLOv5 (You Only Look Once, an object detection algorithm, v5 indicates the fifth version of the algorithm).
[0111] In practice, the computer device stores a deep learning model. The computer device trains the deep learning model based on a set of image samples to obtain the trained result data.
[0112] Optionally, the computer device trains the deep learning model for 300 training rounds. This embodiment of the application does not limit the number of training rounds.
[0113] Step 606: If the precision and recall rates in the result data meet the preset threshold, the trained deep learning model is obtained.
[0114] The training results data includes precision and recall. Precision and recall consist of accuracy and recall.
[0115] In practice, the computer equipment stores a threshold corresponding to the precision and recall rate. When the precision and recall rate in the result data exceeds the preset threshold, the computer equipment stops training the deep learning model, and the trained deep learning model is obtained.
[0116] Optionally, the threshold corresponding to precision and recall is determined by the training requirements. In this embodiment of the application, the specific value of the threshold corresponding to precision and recall is not limited.
[0117] In this embodiment, a deep learning model is trained using an image sample set. When the precision and recall in the trained result data reaches a preset threshold, a trained deep learning model for image detection is obtained, thereby improving the accuracy of image detection.
[0118] In one embodiment, such as Figure 7 As shown, the specific processing steps of step 104 include:
[0119] Step 702: Perform image detection on the image data based on the trained deep learning model to obtain the image detection results.
[0120] In practice, computer equipment performs image detection on image data based on a trained deep learning model to obtain image detection results.
[0121] Step 704: When the image detection result indicates that a target event has occurred on the monitored target road, output the first position coordinates of the pixel of the target event in the image coordinate system.
[0122] In practice, when the image detection result represents a target event occurring on the monitored target road, the computer device outputs the first position coordinates of the pixel of the target event in the image coordinate system.
[0123] In this embodiment, by detecting image data, when the image detection result indicates that a target event has occurred on the monitored target road, the first position coordinates of the pixel of the target event in the image coordinate system are output, which facilitates the subsequent coordinate transformation of the first position coordinates of the target event.
[0124] In one embodiment, such as Figure 8 As shown, the specific processing steps for obtaining the actual location of the installed camera in step 106 include:
[0125] Step 802: Select multiple control points in the image data to obtain the positions of the multiple control points in the image coordinate system.
[0126] In practice, the computer equipment selects multiple control points from the image data, such as... Figure 9As shown. Then, the computer equipment determines the positions of multiple control points in the image coordinate system.
[0127] Optionally, the number of control points selected by the computer device is 100. This embodiment of the application does not limit the number of control points.
[0128] Step 804: Obtain point cloud data of the target road and determine the actual locations of multiple control points in the point cloud data.
[0129] During implementation, computer equipment acquires point cloud data of the target road collected by the data acquisition vehicle. Then, the computer equipment determines the actual positions of multiple control points in the world coordinate system from the point cloud data.
[0130] Step 806: Based on a preset error function, perform data processing on the actual positions of multiple control points and their positions in the image coordinate system to obtain the actual position where the camera is installed.
[0131] In practice, the computer device acquires the initial pose estimate of the camera in the world coordinate system. The computer device stores an error function, as shown in formula (7) below.
[0132]
[0133] In the above formula (7), T * U represents the actual position of the camera in the world coordinate system. i To control the position of the point in the pixel coordinate system, P i Let T be the actual position of the control point in the world coordinate system, T be the initial pose estimate of the camera in the world coordinate system, K be the intrinsic parameter matrix of the camera, n be the number of control points, and i be the i-th control point. The computer device iteratively calculates the positions and actual positions of multiple control points in the image coordinate system according to the error function. When the reprojection error is 1 pixel, the computer device stops the calculation and obtains the actual position of the camera. The expression for the reprojection error is U in the above formula (7). i -KTP.
[0134] Optionally, the computer device stores the actual location where the camera is mounted in a preset storage space.
[0135] In this embodiment, the position of the control point in the image coordinate system and the position of the control point in the world coordinate system are calculated by an error function. When the reprojection error is 1 pixel, the actual position of the camera is obtained, which improves the accuracy of the actual position of the camera and facilitates the coordinate transformation of the second position coordinates of the target event in the camera coordinate system.
[0136] In one embodiment, the specific processing procedure of step 108 includes:
[0137] Step 1002: Construct the camera rotation matrix based on the camera's extrinsic parameters.
[0138] In practice, the computer device constructs the camera rotation matrix based on the camera's extrinsic parameters, as shown in the above formula (5), which will not be repeated here in this embodiment.
[0139] Step 1004: Perform matrix multiplication on the camera's rotation matrix and the second position of the target event in the camera coordinate system. Add the result of the matrix multiplication to the actual position of the camera to obtain the actual position of the target event.
[0140] In practice, the computer device performs matrix multiplication on the second position of the target event in the camera coordinate system according to the following formula (8). Then, the computer device adds the result of the calculation to the actual position of the camera to obtain the actual position of the target event.
[0141]
[0142] In the above formula (8), Let R(qi) be the actual position of the target event, R(qi) be the camera rotation matrix, and Pi be the actual position of the camera. The second position coordinates of the target event in the camera coordinate system.
[0143] Optionally, after obtaining the actual location of the target event, the computer device uploads the actual location of the target event to a cloud server. This application embodiment does not limit the processing methods after obtaining the actual location of the target event.
[0144] In this embodiment, the actual position of the target event is obtained by performing coordinate transformation on the second position of the target event in the camera coordinate system, thereby realizing the localization of the target event.
[0145] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0146] Based on the same inventive concept, this application also provides a target event localization device for implementing the target event localization method described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more target event localization device embodiments provided below can be found in the limitations of the target event localization method described above, and will not be repeated here.
[0147] In one embodiment, such as Figure 11 As shown, a target event localization device is provided, comprising: an acquisition module 1101, a detection module 1102, a construction module 1103, and a transformation module 1104, wherein:
[0148] The acquisition module 1101 is used to acquire image data and camera intrinsic parameters obtained by the camera in real time, and to determine the camera extrinsic parameters based on the image data; the image data represents the road information of the monitored target road.
[0149] The detection module 1102 is used to perform image detection on image data and obtain image detection results. When the image detection results indicate that a target event has occurred on the monitored target road, it outputs the first position coordinates of the pixel representing the target event in the image coordinate system.
[0150] Module 1103 is used to construct the camera's transformation matrix based on intrinsic and extrinsic parameters; and based on the transformation matrix, to convert the first position coordinates of the target event in the image coordinate system into the second position coordinates in the camera coordinate system.
[0151] The transformation module 1104 is used to obtain the actual position of the installed camera, and perform coordinate transformation on the second position coordinates of the target event in the camera coordinate system based on the actual position of the camera to obtain the actual position coordinates of the target event.
[0152] In an exemplary embodiment, the camera's extrinsic parameters include the camera's pitch angle, yaw angle, roll angle, and mounting height. The acquisition module 1101 includes a first acquisition submodule and a determination submodule, wherein the determination submodule includes:
[0153] The segmentation submodule is used to segment lane lines in image data according to a supervised image semantic segmentation algorithm, obtain segmented lane line observations, determine the disappearance point by the intersection of the lane line observations, and obtain the position of the disappearance point in the image coordinate system.
[0154] The first calculation submodule is used to perform matrix operations on the position and intrinsic parameters of the vanishing point in the image coordinate system according to the projection model, so as to obtain the camera's pitch angle and yaw angle.
[0155] The second calculation submodule is used to obtain the actual length of a single lane, and to calculate the pixel length and actual length of the single lane in the image data according to the least squares method to obtain the camera's roll angle and installation height.
[0156] In one exemplary embodiment, before the detection module 1102 performs its operation, the target event localization device further includes:
[0157] The second acquisition submodule is used to acquire the image sample set corresponding to the image data. Each image sample in the image sample set contains information about the position of the pixel point that represents the target event in the image coordinate system.
[0158] The training submodule is used to train a pre-defined deep learning model based on an image sample set to obtain the trained result data.
[0159] The submodule is used to obtain the trained deep learning model when the precision and recall in the result data meet the preset threshold.
[0160] In one exemplary embodiment, the detection module 1102 includes:
[0161] The detection submodule is used to perform image detection on image data based on the trained deep learning model and obtain the image detection results.
[0162] The output submodule is used to output the first position coordinates of the pixel of the target event in the image coordinate system when the image detection result represents a target event occurring on the monitored target road.
[0163] In an exemplary embodiment, the transformation module 1104 includes a third acquisition submodule and a transformation submodule, wherein the third acquisition submodule includes:
[0164] The selection submodule is used to select multiple control points in the image data and obtain the positions of the multiple control points in the image coordinate system.
[0165] The determination submodule is used to acquire point cloud data of the target road and determine the actual positions of multiple control points in the point cloud data.
[0166] The processing submodule is used to process the actual positions and positions in the image coordinate system of multiple control points according to a preset error function to obtain the actual position of the camera.
[0167] In an exemplary embodiment, the transformation module 1104 includes a third acquisition submodule and a transformation submodule, wherein the transformation submodule includes:
[0168] A submodule is built to construct the camera's rotation matrix based on the camera's extrinsic parameters.
[0169] The third operation submodule is used to perform matrix multiplication on the camera's rotation matrix and the second position of the target event in the camera coordinate system. The result of the matrix multiplication is then added to the actual position of the camera to obtain the actual position of the target event.
[0170] Each module in the aforementioned target event location device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0171] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 12 As shown, the computer device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a target event localization method.
[0172] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0173] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0174] The system acquires real-time image data and intrinsic parameters of the camera, and determines extrinsic parameters of the camera based on the image data; the image data represents road information of the monitored target road.
[0175] Image detection is performed on the image data to obtain image detection results. When the image detection results indicate that a target event has occurred on the monitored target road, the first position coordinates of the pixel representing the target event in the image coordinate system are output.
[0176] Based on the intrinsic and extrinsic parameters, a transformation matrix for the camera is constructed; based on the transformation matrix, the first position coordinates of the target event in the image coordinate system are converted into second position coordinates in the camera coordinate system.
[0177] Obtain the actual position of the camera, and based on the actual position of the camera, perform coordinate transformation on the second position coordinates of the target event in the camera coordinate system to obtain the actual position coordinates of the target event.
[0178] In one embodiment, the extrinsic parameters of the camera include the camera's pitch angle, yaw angle, roll angle, and mounting height. When the processor executes the computer program, it also performs the following steps:
[0179] According to the supervised image semantic segmentation algorithm, the lane lines in the image data are segmented to obtain the segmented lane line observation values. The disappearance point is determined by the intersection of the lane line observation values, and the position of the disappearance point in the image coordinate system is obtained.
[0180] Based on the projection model, matrix operations are performed on the position of the disappearance point in the image coordinate system and the intrinsic parameters to obtain the pitch angle and yaw angle of the camera.
[0181] The actual length of a single lane is obtained, and the pixel length of the single lane in the image data and the actual length are calculated using the least squares method to obtain the roll angle and installation height of the camera.
[0182] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0183] Obtain the image sample set corresponding to the image data, wherein each image sample in the image sample set contains information indicating the position of the pixel representing the target event in the image coordinate system;
[0184] The preset deep learning model is trained based on the image sample set to obtain the trained result data.
[0185] If the precision and recall rates contained in the result data meet a preset threshold, a trained deep learning model is obtained.
[0186] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0187] The trained deep learning model is used to perform image detection on the image data to obtain image detection results.
[0188] When the image detection result indicates that a target event has occurred on the monitored target road, the first position coordinates of the pixel of the target event in the image coordinate system are output.
[0189] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0190] Multiple control points are selected from the image data to obtain the positions of the multiple control points in the image coordinate system;
[0191] Obtain point cloud data of the target road, and determine the actual positions of the plurality of control points in the point cloud data;
[0192] Based on a preset error function, the actual positions of the multiple control points and their positions in the image coordinate system are processed to obtain the actual position where the camera is installed.
[0193] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0194] Based on the extrinsic parameters of the camera, construct the rotation matrix of the camera;
[0195] A matrix multiplication operation is performed on the rotation matrix of the camera and the second position of the target event in the camera coordinate system. The result of the matrix multiplication operation is then added to the actual position of the camera to obtain the actual position of the target event.
[0196] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0197] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0198] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0199] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0200] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0201] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for locating a target event, characterized in that, The method includes: The system acquires real-time image data and intrinsic parameters of the camera, and determines extrinsic parameters of the camera based on the image data; the image data represents road information of the monitored target road. Image detection is performed on the image data to obtain image detection results. When the image detection results indicate that a target event has occurred on the monitored target road, the first position coordinates of the pixel representing the target event in the image coordinate system are output. The extrinsic parameters include the camera's pitch angle, yaw angle, roll angle, and installation height. Based on the intrinsic and extrinsic parameters, a transformation matrix for the camera is constructed; based on the transformation matrix, the first position coordinates of the target event in the image coordinate system are transformed into a second position coordinate in the camera coordinate system; the transformation matrix is... ,in, The installation height of the camera, , , The pitch angle is... , , The yaw angle is... The focal length is one of the intrinsic parameters of the camera. The camera optical center is one of the intrinsic parameters of the camera. The transformation matrix is... The actual position of the camera is obtained, and the second position coordinates of the target event in the camera coordinate system are transformed according to the actual position of the camera to obtain the actual position coordinates of the target event. The step of converting the first position coordinates of the target event in the image coordinate system to the second position coordinates in the camera coordinate system according to the transformation matrix includes: based on A matrix operation is performed on the transformation matrix and the first position coordinates of the target event in the image coordinate system to obtain the second position coordinates of the target event in the camera coordinate system; wherein, Let be the transformation matrix. Let be the second position coordinates of the target event in the camera coordinate system. The coordinates of the first position of the target event in the image coordinate system.
2. The method according to claim 1, characterized in that, The camera's extrinsic parameters include the camera's pitch angle, yaw angle, roll angle, and mounting height; determining the camera's extrinsic parameters based on the image data includes: According to the supervised image semantic segmentation algorithm, the lane lines in the image data are segmented to obtain the segmented lane line observation values. The disappearance point is determined by the intersection of the lane line observation values, and the position of the disappearance point in the image coordinate system is obtained. Based on the projection model, matrix operations are performed on the position of the disappearance point in the image coordinate system and the intrinsic parameters to obtain the pitch angle and yaw angle of the camera. The actual length of a single lane is obtained, and the pixel length of the single lane in the image data and the actual length are calculated using the least squares method to obtain the roll angle and installation height of the camera.
3. The method according to claim 1, characterized in that, Before performing image detection on the image data to obtain an image detection result, and when the image detection result indicates that a target event has occurred on the monitored target road, outputting the first position coordinates of the pixel representing the target event in the image coordinate system, the following steps are included: Obtain the image sample set corresponding to the image data, wherein each image sample in the image sample set contains information indicating the position of the pixel representing the target event in the image coordinate system; The preset deep learning model is trained based on the image sample set to obtain the trained result data. If the precision and recall rates contained in the result data meet a preset threshold, a trained deep learning model is obtained.
4. The method according to claim 3, characterized in that, The image detection process is performed on the image data to obtain an image detection result. When the image detection result indicates that a target event has occurred on the monitored target road, the first position coordinates of the pixel representing the target event in the image coordinate system are output, including: The trained deep learning model is used to perform image detection on the image data to obtain image detection results. When the image detection result indicates that a target event has occurred on the monitored target road, the first position coordinates of the pixel of the target event in the image coordinate system are output.
5. The method according to claim 1, characterized in that, The step of obtaining the actual location where the camera is installed includes: Multiple control points are selected from the image data to obtain the positions of the multiple control points in the image coordinate system; Obtain point cloud data of the target road, and determine the actual positions of the plurality of control points in the point cloud data; Based on a preset error function, the actual positions of the multiple control points and their positions in the image coordinate system are processed to obtain the actual position where the camera is installed.
6. The method according to claim 1, characterized in that, The step of obtaining the actual position of the camera and, based on the actual position of the camera, performing a coordinate transformation on the second position of the target event in the camera coordinate system to obtain the actual position of the target event includes: Based on the extrinsic parameters of the camera, construct the rotation matrix of the camera; A matrix multiplication operation is performed on the rotation matrix of the camera and the second position of the target event in the camera coordinate system. The result of the matrix multiplication operation is then added to the actual position of the camera to obtain the actual position of the target event.
7. A target event localization device, characterized in that, The device includes: The acquisition module is used to acquire image data and intrinsic parameters of the camera in real time, and determine the extrinsic parameters of the camera based on the image data; the image data represents the road information of the monitored target road. The detection module is used to perform image detection on the image data to obtain image detection results. When the image detection results indicate that a target event has occurred on the monitored target road, the module outputs the first position coordinates of the pixel representing the target event in the image coordinate system. The extrinsic parameters include the camera's pitch angle, yaw angle, roll angle, and installation height. The construction module is used to construct the camera's transformation matrix based on the intrinsic and extrinsic parameters; and to convert the target event's first position coordinates in the image coordinate system to a second position coordinate in the camera coordinate system based on the transformation matrix; the transformation matrix is... ,in, The installation height of the camera, , , The pitch angle is... , , The yaw angle is... The focal length is one of the intrinsic parameters of the camera. The camera optical center is one of the intrinsic parameters of the camera. The transformation matrix is... The transformation module is used to obtain the actual position of the camera, and perform coordinate transformation on the second position coordinates of the target event in the camera coordinate system according to the actual position of the camera to obtain the actual position coordinates of the target event; The building module is specifically used for... A matrix operation is performed on the transformation matrix and the first position coordinates of the target event in the image coordinate system to obtain the second position coordinates of the target event in the camera coordinate system; wherein, Let be the transformation matrix. Let be the second position coordinates of the target event in the camera coordinate system. The coordinates of the first position of the target event in the image coordinate system.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.