Multi-task joint execution method and device, electronic equipment and readable storage medium
Through drone image feature extraction and convolutional neural network repair, combining road feature association and traffic participant tracking, the problem of high computing power demand when drones acquire ground information is solved, and the efficiency of image repair and topological map construction is improved and accuracy improve user experience.
Patent Information
- Application Number
- CN202411082068.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-12-06
AI Technical Summary
In the prior art, when obtaining ground information through drones, the computing power demand is high, resulting in low efficiency and poor user experience. Aerial images are easily affected by shooting angles, lighting and weather, resulting in content distortion, ground target detection and recognition difficulty.
Using a multi-task joint execution method, image features are extracted from the original image through preset convolutional neural network, image repair, road topology map is constructed, and traffic participants are tracked to realize image repair, road topology map expansion and target tracking. multitasking reduces the demand for computing power and improves efficiency.
It realizes the stable performance of the UAV performing tasks in uneven lighting and cloudy weather, improves image repair quality and road topology accuracy, enhances the accuracy and efficiency of traffic participants tracking, and reduces the multi-task computing Resource requirements.
Smart Images

Figure CN119091295A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method, device, electronic device and readable storage medium for jointly executing multiple tasks. Background Art
[0002] With the development of drone technology, the application of small low-altitude drones has attracted widespread attention. Collecting ground information through drones is an important field of application of drone technology, among which drone remote sensing images are one way to collect ground information.
[0003] In the existing technology, vehicle-mounted drone technology is used to provide car cabins with functions such as on-board photography and simultaneous image transmission. It can also sense road conditions, detect traffic flows, and intelligently analyze vehicle behavior to perform intelligent traffic tasks through aerial photography. However, obtaining the required information through drones requires high computing power, resulting in low efficiency and poor user experience. Summary of the invention
[0004] In view of this, the embodiments of the present application provide a multi-task joint execution method, device, electronic device and readable storage medium to solve the problem in the prior art that obtaining the required information through drones requires high computing power, resulting in low efficiency.
[0005] According to a first aspect of an embodiment of the present application, a method for jointly executing multiple tasks is provided, including:
[0006] Acquire original images of multiple adjacent frames;
[0007] Extract image features from the original image through a preset convolutional neural network, and repair the original image based on the image features to obtain a repaired image;
[0008] Extract road features from each frame of the restored image, associate the road features corresponding to adjacent frames, obtain the association relationship between the road features of adjacent frames, and construct a road topology map in real time based on the association relationship;
[0009] Based on the image features, the target detection frame corresponding to the traffic participant in the repaired image is corrected to track the traffic participant.
[0010] A second aspect of an embodiment of the present application provides a multi-task joint execution device, including:
[0011] An acquisition module is configured to acquire original images of a plurality of adjacent frames;
[0012] The restoration module is configured to extract image features from the original image through a preset convolutional neural network, and restore the original image based on the image features to obtain a restored image;
[0013] The road processing module is configured to extract road features from each frame of the repaired image, associate the road features corresponding to adjacent frames, obtain the association relationship between the road features of adjacent frames, and construct a road topology map in real time based on the association relationship;
[0014] The tracking module is configured to correct the target detection frame corresponding to the traffic participant in the repaired image based on the image features to track the traffic participant.
[0015] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0016] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0017] Compared with the prior art, the embodiments of the present application have the following beneficial effects: obtaining original images of multiple adjacent frames, extracting image features from the original images through a preset convolutional neural network, repairing the original images according to the image features to obtain repaired images, extracting road features from the repaired images of each frame, associating the road features corresponding to the adjacent frames respectively, thereby obtaining the association relationship between the adjacent frame images, and constructing a road topology map in real time according to the association relationship, and correcting the target detection frames corresponding to the traffic participants in the repaired images according to the image features, so as to achieve tracking of all traffic participants according to the repaired images, that is, the original images can be repaired according to the image features extracted by a preset convolutional neural network, and the road topology map can be obtained according to the original images, and the traffic participants can also be tracked according to the repaired images and the image features, so that at least three tasks of image repair, road topology map expansion and target tracking can be completed by extracting image features from the original image once, without the need to extract image features separately according to the requirements of each task in the prior art, which can reduce the demand for computing power during multi-task calculations, save resources, and thus improve task processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 It is a flowchart of a method for joint execution of multiple tasks provided in an embodiment of the present application;
[0020] Figure 2 It is a flowchart of another multi-task joint execution method provided in an embodiment of the present application;
[0021] Figure 3 is a schematic diagram of a multi-task joint execution device provided in an embodiment of the present application;
[0022] Figure 4 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0023] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0024] A method and device for joint execution of multiple tasks according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.
[0025] In the existing technology, vehicle-mounted drones can provide vehicle intelligent cockpit systems with functions such as on-board shooting and image transmission, providing emotional value for users' travel experience. In addition, vehicle-mounted drones have shown strong application potential in intelligent traffic tasks such as road perception, traffic flow detection, and intelligent analysis of vehicle behavior. In the technical development trend of decoupling autonomous driving technology from high-precision maps, a low-cost method for obtaining real-time traffic information beyond visual range is provided for autonomous driving systems. However, the following problems still exist in the process of processing aerial images:
[0026] When processing aerial images, the traffic extraction task is usually divided into multiple independent subtasks without considering the deep correlation between each subtask, which affects the effect of traffic information processing. In addition, the processing of multiple tasks requires a lot of computing power, which easily affects the real-time performance of traffic information extraction.
[0027] Aerial images are easily affected by the shooting angle of the drone, light intensity and external weather, which may lead to distortion of the aerial image content and irregular images of ground roads and vehicles.
[0028] Due to the limitation of drone shooting distance, the ground targets shown in aerial images are relatively small, and the background is relatively complex and obstructed, which hinders the detection and identification of target information of traffic target objects.
[0029] In view of this, the present application proposes a method for the joint execution of multiple tasks, wherein image features are required during the execution of each task, which can be achieved by extracting features from the original image once. When the application scenario is a drone, the image features obtained by extracting features from the original image once can achieve the purpose of repairing the image, constructing a more detailed road topology map, and tracking traffic participants through the joint execution of multiple tasks, so that the tasks performed by the drone show more stable performance under uneven lighting and cloudy and rainy weather conditions, specifically:
[0030] Drone image restoration: Due to the movement or shaking of the drone, the original image obtained by the drone is blurred or has residual images. This application uses historical frames to guide the current frame based on the continuity of the drone's movement to eliminate the blur and residual images of the current frame.
[0031] Constructing a road topology map: Extracting road features from the repaired image and the correlation between road features in adjacent frames. At the same time, determining the phase difference of the drone between adjacent frames. Based on the phase difference and correlation, constructing a road topology map in real time.
[0032] Target tracking: Based on the image features, the target detection frame corresponding to the traffic participants in the repaired image is corrected. In order to improve the detection accuracy of a large number of small targets, the relative motion in the repaired image can be corrected through a motion compensation algorithm, and then the coordinates of the misdetected or occluded targets are inferred based on the temporal context information, thereby completing the tracking of the traffic participants.
[0033] Figure 1 1 is a flowchart of a method for joint execution of multiple tasks provided in an embodiment of the present application. The method for joint execution of multiple tasks can be executed by a controller in the cloud or in a target vehicle. Figure 1 As shown, the multi-task joint execution method includes the following steps:
[0034] S101, acquiring original images of multiple adjacent frames;
[0035] Specifically, multiple adjacent frames of original images are continuously captured from the video stream, that is, video data in a video file or a video stream acquired in real time can be converted into multiple frames of original images that can be processed.
[0036] Among them, since each frame of the original image is updated and transmitted in the form of a video stream, there is a temporal correlation between the original images of multiple adjacent frames, and there is an internal spatial information correlation between the original images of a single frame.
[0037] Alternatively, you can use a video processing library, such as the Open Source Computer Vision Library (OpenCV), to read video files or capture the output of a camera.
[0038] S102, extracting image features from the original image by using a preset convolutional neural network, and repairing the original image based on the image features to obtain a repaired image;
[0039] Specifically, in order to improve the image quality, image features can be extracted from the original image through a preset convolutional neural network, and the image features can capture information such as texture, edge, shape, etc. of the image, providing a basis for subsequent image processing and image restoration. The preset convolutional neural network can be any neural network that can extract image features in the prior art.
[0040] In addition, the original image is repaired according to the image features to remove blur, residual image, noise and other contents in the original image. Among them, when the shooting device for obtaining the original image in this application is a vehicle-mounted drone, the blur and residual image in the original images of multiple adjacent frames obtained by the drone are a small part of the overall image. Masks and attention mechanisms can be used to limit the blur, residual image and other information to the local area of the original image, and the original image is repaired through the diffusion model to achieve more accurate image repair and obtain a repaired image.
[0041] In addition, the repaired image obtained according to the image features can be clearer and more complete in visual effect, which is convenient for subsequent analysis and processing based on the repaired image.
[0042] S103, extracting road features from each frame of the restored image, correlating the road features corresponding to adjacent frames, obtaining the correlation relationship between the road features of adjacent frames, and constructing a road topology map in real time based on the correlation relationship;
[0043] Specifically, after obtaining each frame of the repaired image, the repaired image can be classified into two categories to obtain a binary classification map, that is, the road and background are segmented. For the segmented repaired image, the road features in the repaired image can be extracted. The road features facilitate the understanding of the traffic scene. The road features include key point features and key line features. The road features can be extracted through a semantic segmentation model.
[0044] In addition, by associating the road features corresponding to each frame in the adjacent frames, the association relationship between the road features in the adjacent frames can be obtained, and the association relationship between the adjacent frame images can be obtained, thereby determining the continuity of the road structure between the adjacent frame images.
[0045] In addition, the road topology map can be constructed in real time based on the association relationship. The road topology map is a way of abstractly representing the key points corresponding to the key point features and the key lines corresponding to the key line features. The key lines constitute the connection relationship between the key points, showing the connection relationship, directionality and structural characteristics between the roads. The road topology map can be constructed and expanded according to the real-time original image to achieve the purpose of real-time iterative road topology map, so as to extract the roads in the original image in real time.
[0046] S104, based on the image features, modifying the target detection frame corresponding to the traffic participant in the repaired image to track the traffic participant.
[0047] Specifically, in video processing, the detection and tracking of traffic participants is an important link. The target detection algorithm can be used to identify traffic participants in the repaired image and generate an initial target detection frame. The target detection algorithm can be a deep generative network regional convolutional neural network (Faster Rich featurehierarchies for accurate object detection and semantic segmentation, Faster R-CNN) based on deep learning, etc. Traffic participants include pedestrians, vehicles, etc.
[0048] In addition, due to factors such as the quality, occlusion, and lighting of the repaired image, the initial target detection frame may not be accurate enough. In order to improve the accuracy and robustness of target tracking, the initial target detection frame can be corrected through image features. The correction can be achieved by comparing the features of the area around the initial target detection frame with the features in the target feature library to obtain the target detection frame, so that it can more accurately select traffic participants.
[0049] In addition, traffic participants can be tracked between consecutive frames based on the target detection frame. This can be achieved by using the Hungarian algorithm, Kalman filtering, etc. Motion compensation is performed based on the original images of adjacent frames, and the position and size of the initial target detection frame are adjusted to maintain the identity consistency of traffic participants between consecutive frame images, and their motion trajectories are estimated to track traffic participants.
[0050] According to the technical solution provided in the embodiment of the present application, the original image can be repaired according to the image features extracted by a preset convolutional neural network, and a road topology map can be obtained according to the original image. Traffic participants can also be tracked according to the repaired image and the obtained image features. That is, the image features extracted once from the original image by the preset convolutional neural network can be used to perform at least three tasks of image restoration, road topology and target tracking, which can reduce the demand for computing power during multi-task calculations and save resources.
[0051] In some embodiments, road features are extracted from each frame of the restored image, road features corresponding to adjacent frames are associated, the association relationship between the road features of adjacent frames is obtained, and a road topology map is constructed in real time based on the association relationship, including:
[0052] The inpainted image of each frame is divided into two categories by a preset image segmentation model to obtain a two-category image; the two-category image is used to segment the road and the background other than the road;
[0053] The road features corresponding to each frame are extracted from the binary classification images corresponding to each frame, and the road features include key point features and line features corresponding to the lines between the key points;
[0054] Determine the correlation relationship between the road features of the previous frame and the road features of the current frame in the adjacent frames, and determine the phase difference between the restored images of the two adjacent frames based on the correlation relationship;
[0055] A road topology map is constructed based on road features corresponding to adjacent frames, a phase difference between restored images of two adjacent frames, and positioning parameters of a shooting device, wherein the original image is obtained by shooting the device.
[0056] Specifically, each frame of the repaired image is divided into two categories through a preset image segmentation model, wherein the preset image segmentation model can classify the pixels in the repaired image into two categories, one category represents pixels in the road area, and the other category is all backgrounds except the road, such as buildings, vegetation, sky, etc. The binary classification is based on pixel level recognition and has a high accuracy rate.
[0057] In addition, after obtaining the binary classification image, the road features corresponding to each frame are extracted from the binary classification image corresponding to each frame. The road features mainly include key point features and line features corresponding to the connecting lines between key points.
[0058] Among them, key point features are used to describe key road nodes, such as road intersections, turning points, or points with significant shape changes. Key points can be located through image analysis and morphology to determine key point features;
[0059] Keyline features are used to describe the lines between key points, that is, the actual direction of the road. The line features can be identified and extracted through the line detection algorithm in image processing, thereby determining the basic skeleton of the road. The line detection algorithm can be Hough transform, straight line fitting after edge detection, etc.
[0060] In addition, in order to construct a continuous and consistent road topology map, it is necessary to determine the correlation relationship between road features between adjacent frames. The correlation relationship between the road features of the previous frame in the adjacent frames and the road features of the current frame can be determined by feature matching, such as matching based on feature descriptors, or using temporal continuity to track the motion trajectories of key points and key lines, thereby identifying the corresponding positions of the road features of the previous frame in the adjacent frames in the current frame, thereby establishing an association relationship.
[0061] In addition, based on the correlation between the road features of adjacent frames, the phase difference between the restored images of two adjacent frames can be determined. The phase difference reflects the change in perspective caused by the movement of the shooting device during continuous shooting. The change in perspective can affect the local features of the restored image and also affect the construction of the overall road topology map. By determining the phase difference through the correlation relationship, the image deformation caused by the movement can be corrected to ensure the continuity of the road topology map in space and time.
[0062] In addition, a road topology map is constructed in real time based on the road features corresponding to adjacent frames, the phase difference between the repaired images of two adjacent frames, and the positioning parameters of the shooting equipment. The positioning parameters include coordinates and direction angles obtained according to the global satellite positioning system, which can locate the road features in the geographic coordinate system and adjust them through the phase difference of the associated relationship, so that the road topology map can accurately reflect the layout and direction of the actual road network. The positioning parameters can also improve the accuracy of the road topology map, where the original image is taken by the shooting equipment.
[0063] According to the technical solution provided in the embodiment of the present application, the corresponding road features can be extracted from the repaired images of each frame, and the road features of adjacent frames can be associated to obtain the correlation relationship between the road features of the previous frame and the road features of the current frame in the adjacent frames. The correlation relationship shows a certain spatiality, so that a more accurate road topology map can be constructed.
[0064] In some embodiments, determining the association relationship between the road features of the previous frame and the road features of the current frame in adjacent frames, and determining the phase difference between the restored images of two adjacent frames based on the association relationship, includes:
[0065] Determine the measurement value of the nth line feature and the measurement value of the jth key point feature in the current frame, and determine the mth line feature corresponding to the nth line feature and the ith key point feature corresponding to the jth key point feature in the previous frame;
[0066] Based on the uncertainty of line feature extraction and the measurement value of the nth line feature in the current frame, a first association relationship between the nth line feature of the current frame and the mth line feature of the previous frame is established;
[0067] Based on the uncertainty of point feature extraction and the measurement value of the jth key point feature in the current frame, a second association relationship between the jth point feature of the current frame and the ith point feature of the previous frame is established;
[0068] Based on the first association relationship, the second association relationship and the posture of the shooting device when measuring the measurement value of the nth line feature, the phase difference between adjacent frame repaired images is determined.
[0069] Specifically, the line feature is the feature of the line corresponding to the line between the key points. The measured value of the line feature may include the coordinates of the starting point and the end point of the key line in the repaired image and possible slope or curvature information. The measured value of the point feature may include the precise coordinates of the key point in the repaired image. The values of n, j, m and i are all greater than or equal to 1.
[0070] Furthermore, since determining the correlation between road features between adjacent frames depends on the accurate recognition of significant features in the image, the position and posture of the camera device can also be recorded when the camera device is shooting to achieve the continuity modeling of road features in different frames, that is, the measurement value of the nth line feature and the measurement value of the jth key point feature are both measured when the camera device is in the position x. t The pose represents the position and orientation in three-dimensional space.
[0071] Due to factors such as measurement errors and environmental interference, the positions of the two line features corresponding to the above-mentioned adjacent frames in the repaired image will not be completely consistent. Therefore, the uncertainty of deep learning image line feature extraction can be represented by a first preset error value. The first preset error value can be a fixed value set based on experience or statistical data, or it can be obtained through multiple experiments and dynamic adjustments. Then, the first preset error value is used to establish a first correlation relationship between the nth line feature of the current frame and the mth line feature of the previous frame.
[0072] In addition, similarly, for the j-th key point feature of the current frame, the uncertainty of deep learning image point feature extraction is represented by a second preset error value, and then a second association relationship between the j-th point feature of the current frame and the i-th point feature of the previous frame is established by the second preset error value.
[0073] In addition, the phase difference between adjacent frame repaired images is further determined based on the first association relationship, the second association relationship, and the posture of the shooting device when measuring the measurement value of the nth line feature.
[0074] Among them, the relative translation and rotation, that is, the phase difference, between two adjacent frames can be determined through geometric transformation and error minimization multiplication using the posture information related to the current frame and the nth line feature, as well as the position of the mth line feature in the previous frame. Similarly, auxiliary calculation can be performed through the second association relationship corresponding to the key point features to improve the accuracy and robustness of the phase difference estimation.
[0075] In one embodiment of the present application, an image segmentation network model can be used to divide the restoration image corresponding to the previous frame in the input adjacent frames into a binary classification map. For the binary classification map, key road pixel clusters are fitted to extract road features in the restoration image. The road features can be expressed as N = {m p ,m l}, where m p represents the key point feature, m l Represents a keyline feature.
[0076] The above steps are performed on the repaired image of the current frame to obtain the road features of the adjacent frames. The correlation relationship between the road features of the previous frame and the road features of the current frame in the adjacent frames can be defined as c = [c p ,c l ], where c p ={i,j} represents the second correlation relationship between the i-th key point feature in the road feature of the previous frame and the j-th key point feature in the road feature of the current frame. Similarly, c l ={m,n} is used to represent the first association relationship between line features in adjacent frame images, that is, it represents the first association relationship between the m line features in the road features of the previous frame and the nth key line feature in the road features of the current frame.
[0077] The first correlation relationship between the line features corresponding to the previous frame restoration image and the line features corresponding to the current frame restoration image can be expressed by the following formula:
[0078] in, Represents the mth line feature of the previous frame repaired image, (m,n)∈c l Indicates the corresponding relationship of line features, n and m are both greater than or equal to 1, Indicates the position of the camera device is x t The measured value of the nth line feature in the case of Represents a first preset error value, which is used to represent the uncertainty of line feature extraction.
[0079] Similarly, the second correlation relationship between the key point features corresponding to the previous frame repair image and the key point features corresponding to the current frame repair image can be expressed by the following formula:
[0080] in, Represents the i-th key point feature of the previous frame repaired image, {i,j}∈c p Indicates the corresponding relationship between key point features, i and j are both greater than or equal to 1, Indicates the position of the camera device is x t The measured value of the jth key point feature in the case of Represents the second preset error value, which is used to represent the uncertainty of point feature extraction. It should be noted that the previous frame is relative to the current frame, that is, the previous frame is the previous frame of the current frame in terms of time sequence; for example, assuming there are five frames of images from 1 to 5, if the current frame is the 5th frame, the previous frame is the 4th frame; if the current frame is the predicted 6th frame, the previous frame is the 5th frame.
[0081] Continuing with the above example, the phase difference can be obtained by maximum likelihood estimation based on the first association relationship and the second association relationship corresponding to the current frame and the posture of the camera when measuring the measurement value of the nth line feature, where the maximum likelihood estimation is expressed as:
[0082]
[0083] in, Represents the phase difference between two adjacent frame repaired images, x t Indicates the pose of the camera corresponding to the current frame. represents the maximum likelihood function, that is, the function that obtains the minimum parameter value in its domain, where Z t Indicates road characteristics. represents the i-th key point feature of the previous frame, Represents the mth line feature of the previous frame.
[0084] According to the technical solution provided in the embodiment of the present application, the first association relationship and the second association relationship of the road features in adjacent frames can be determined, and the posture of the shooting device corresponding to the previous frame in the adjacent frames can be determined. According to the first association relationship, the second association relationship and the corresponding posture, the phase difference between the repaired images of adjacent frames can be obtained. The phase difference can be used to construct and update the road topology map, determine the running direction of the shooting device, perform image stitching, video stabilization, scene reconstruction, etc.
[0085] In some embodiments, constructing a road topology map based on road features corresponding to adjacent frames, a phase difference between two adjacent frame restoration images, and positioning parameters of a shooting device includes:
[0086] Determine the key point features corresponding to the previous frame as a first vertex set, determine the line features corresponding to the previous frame as a first edge set, and construct a road topology graph to be updated based on the first vertex set and the first edge set;
[0087] Determine the key point features corresponding to the current frame as a second vertex set, determine the line features corresponding to the current frame as a second edge set, and construct a reference road topology map based on the second vertex set and the second edge set;
[0088] Determine the estimated position and distance of the displacement of the photographing device based on the phase difference between the repaired images of two adjacent frames and the positioning parameters of the photographing device;
[0089] Based on the estimated orientation and distance of the displacement of the photographing device, the road topology map to be updated is compared with the reference road topology map in the direction corresponding to the estimated orientation, and a vertex satisfying an update condition is determined as an update vertex, wherein the update condition includes being in the direction corresponding to the estimated orientation, not being included in the first vertex set and being included in the second vertex set;
[0090] The first vertex set and the first edge set are updated based on the updated vertices, and the road topology graph to be updated is updated based on the updated first vertex set and the first edge set to obtain a road topology graph.
[0091] Specifically, the key point features corresponding to the previous frame in the adjacent frames are determined as the first vertex set, which represents the key positions in the road. At the same time, the detected line features are used as the edges connecting the vertices in the first vertex set, thereby forming a first edge set. The road topology map to be updated is constructed based on the first vertex set and the first edge set. Similarly, the key point features corresponding to the current frame are determined as the second vertex set, and the line features corresponding to the current frame are determined as the second edge set. A reference road topology map is constructed based on the second vertex set and the second edge set. The reference road topology map can be used as a comparison object for comparing with the road topology map to be updated, thereby updating the road topology map to be updated to obtain the road topology map.
[0092] Among them, the road topology map is a quantitative map used to represent the relative position relationship and connection relationship between the elements in the road network. It has the transitional characteristics between maps and statistical graphs. In the road topology map, the intersections and endpoints of the roads are abstracted as nodes, and the roads themselves are abstracted as line segments or branches connecting the nodes. The nodes and line segments together constitute the road topology map, which can reflect the connection relationship and spatial layout between roads, that is, the road topology map can reflect the direction and length of each line feature. It should be noted that when the road topology map is abstracted, the width of the road topology map can be set to 1. Of course, the width value of the road topology map is not specifically limited here.
[0093] Among them, for the updated road topology map and the reference road topology map, the data format needs to be kept consistent. The road topology map may include road names, intersection names, coordinate systems, attribute fields, etc.
[0094] In addition, filtering algorithms such as extended Kalman filtering and particle filtering can be used to determine the estimated position and distance of the displacement of the shooting device by comparing the phase difference between the repaired images of two adjacent frames and the positioning parameters of the shooting device. The comparison can be done through timestamps or video encoding information. Among them, the displacement, speed and direction change of the shooting device between two adjacent frames can be estimated through the positioning parameters, which is conducive to accurately judging the moving track of the shooting device, thereby facilitating the determination of the estimated direction and distance of the reference road topology map relative to the road topology map to be updated. Then, the estimated position and distance are applied to the road topology expansion.
[0095] In addition, the estimated azimuth and distance of the displacement of the shooting device are determined based on the phase difference and the positioning parameters. Based on the estimated azimuth and distance of the displacement of the shooting device, the road topology map to be updated and the reference road topology map are compared in the direction corresponding to the estimated azimuth, and the vertices that meet the update conditions are determined as update vertices, wherein the update conditions include being in the direction corresponding to the estimated azimuth, not included in the first vertex set, and included in the second vertex set. That is, the vertices that do not exist in the previous frame of two adjacent frames but are newly added in the current frame and in the direction corresponding to the estimated azimuth are determined as update vertices.
[0096] For example, in the first vertex set V = {v1, v2}, that is, the first vertex set includes two key point features v1 and v2, and in the second vertex set V′ = {v2, v3}, that is, the second vertex set includes two key point features v2 and v3; assuming that v3 is in the estimated orientation, it can be seen that v3 is not in the previous frame of the two adjacent frames, that is, Therefore, v3 is updated as the update vertex to the first vertex set, that is, the general formula of the update condition can be expressed as Where V represents the vertex set of the road topology graph to be updated, V′ represents the vertex set of the reference road topology graph, v1 represents the first vertex in the road topology graph, v2, v3, v i And so on for the second vertex, the third vertex, and the i-th vertex in the road topology graph.
[0097] Furthermore, other factors may be added as update conditions, such as connectivity with an existing road topology map, so as to ensure that the updated road topology map is accurate and coherent.
[0098] In addition, the path between the updated vertex and the vertex in the first vertex set is abstracted as an edge e i , and e i The first edge set is added to update the first edge set, and the road topology graph to be updated is updated according to the updated first vertex set and the first edge set to obtain a road topology graph.
[0099] Among them, since the road topology map to be updated is the road topology map corresponding to the previous frame in the adjacent frames, it includes all the data of the previous frame in the adjacent frames, and the reference road topology map may not include part of the data of the previous frame in the adjacent frames. Therefore, the road topology map obtained by updating the road topology map to be updated according to the reference road topology map can display in real time all the road information corresponding to the original image collected from the beginning, thereby realizing that as the shooting equipment continues to move, the road topology map will be continuously explored and iterated, and the flexibility of road expansion is improved.
[0100] Furthermore, the updated road topology map can be smoothed to eliminate noise or inconsistencies, improve the accuracy and readability of the road topology map, and can also be verified based on the road topology map generated by actual road data, historical records or other data sources, and optimized and adjusted as needed.
[0101] According to the technical solution provided in the embodiment of the present application, the reference road topology map can be used as supplementary data for the road topology map to be updated, and the road topology map to be updated is continuously iterated and expanded as the shooting device moves, and finally a road topology map with complete data is obtained, thereby realizing the real-time road extraction function based on vision of the shooting device.
[0102] In some embodiments, based on the image features, the target detection frame corresponding to the traffic participant in the repaired image is corrected to track the traffic participant, including:
[0103] Based on the target tracking algorithm, the traffic participants in the repaired image of each frame are detected to obtain the target detection frame corresponding to each frame;
[0104] A plurality of target point features are randomly sampled on the image features corresponding to each frame, and the target point features corresponding to each frame are tracked in the subsequent frames of each frame to determine the target point coordinates corresponding to each target point feature in each frame;
[0105] Determine an affine transformation matrix based on an offset between target point coordinates of the same target point feature in a previous frame and a current frame;
[0106] Based on the affine transformation matrix, the coordinates of each target point corresponding to the previous frame are corrected and transformed to obtain the corrected coordinates of the target point in the current frame;
[0107] The target detection frame of the current frame is predicted based on the corrected target point coordinates of the current frame, and the intersection-and-union ratio between the detected target detection frame of the current frame and the predicted target detection frame of the current frame is determined; and the traffic participant is tracked based on the intersection-and-union ratio.
[0108] Specifically, based on the extracted image features and the repaired image, the target detection frame of the traffic participants in the repaired image is accurately corrected and tracked, which can improve the accuracy and efficiency of tracking the traffic participants. The target detection algorithm can detect each frame of the repaired image to identify and locate the traffic participants in the repaired image, wherein the target detection algorithm can generate a target detection frame for each detected traffic participant based on the color, texture, shape and other features in the repaired image, which is used to mark the position and size of each target participant in the repaired image.
[0109] In addition, multiple target point features are randomly collected on the image features corresponding to each frame of the repaired image, and the collected target point features are tracked in subsequent frames, wherein the target point features can be corner points, edge points or significantly changed pixels of the target detection frame, etc., which can represent the key structural information of traffic participants. The target point features can be tracked between consecutive frames by sparse optical flow method, feature point matching or deep learning feature tracking algorithm, so as to determine the target point coordinates corresponding to each target point feature in each frame.
[0110] For example, when the shooting device is a drone, the target tracking algorithm can mainly adopt a vision-based target tracking algorithm. This type of algorithm mainly relies on the overlap of the target detection result boxes in each repaired image frame in a short period of time, and associates the target identity according to the overlap. Therefore, the target detection algorithm can be executed in each repaired image frame to detect traffic participants. Since the drone obtains the original image through aerial photography, the viewing angle of the original image is far and the target on the original image is small. The target detection network of the Transformer model can be used, and the diffusion model can be combined to reduce noise and improve the detection accuracy of traffic participants.
[0111] In addition, since the camera of the shooting device is dynamic when acquiring the original image, the position of the detection frame will change dramatically, which may cause the position distance of the same traffic participant in the repaired image of adjacent frames to be greatly different, resulting in the conversion of the target detection frame or erroneous negative examples. The motion mode corresponding to the camera of the shooting device can be divided into two types, namely rigid motion and non-rigid motion. Rigid motion can be understood as the change of the posture of the shooting device, and non-rigid motion can be understood as the movement of traffic participants (such as pedestrians) in the image.
[0112] In the absence of additional information about the camera motion of the shooting device or the intrinsic matrix of the camera, projecting the rigid motion of the camera onto the image screen based on the repaired image registration of adjacent frames is an approximate method, and the background motion deviation of the repaired image can be corrected by motion compensation. Specifically: According to the position offset of the target point features between each frame, the local transformation relationship between the target participating point from the previous frame to the current frame between adjacent frames can be obtained. The affine transformation matrix is determined according to the offset of the target point coordinates corresponding to each target point feature in each frame through the least squares method or other algorithms. The affine transformation matrix describes the geometric transformation of traffic participants between adjacent frames due to factors such as motion, perspective change or image evenness. Among them, the affine transformation matrix includes transformation parameters such as translation, rotation, and scaling, which can more accurately describe the motion pattern of each traffic participant.
[0113] In addition, through the target point coordinates of the previous frame and the affine transformation matrix, the coordinates of each target point corresponding to the previous frame can be corrected and transformed to obtain the corrected target point coordinates of the current frame, thereby predicting the target detection frame of the current frame and determining the intersection over union (IoU) between the detected target detection frame of the current frame and the predicted target detection frame of the current frame. Among them, IoU can describe the index of the overlap between the detected target detection frame and the predicted target detection frame. The higher the ratio of the intersection over union, the higher the similarity between the two target detection frames, which reflects the accuracy and stability of target tracking.
[0114] In addition, based on the obtained IoU value, the continuity and accuracy of target tracking can be judged. For example, when the IoU value exceeds the preset threshold, the tracking of the current frame can be considered successful, and the target detection box in the next frame can be predicted and tracked; when the IoU value is lower than the preset threshold, the parameters are adjusted or other strategies are adopted to ensure target tracking.
[0115] According to the technical solution provided in the embodiment of the present application, by calculating the intersection-and-union ratio between the detected target detection frame of the current frame and the predicted target detection frame of the current frame, that is, by motion compensation, the motion distance of the image background in the repaired image is determined by the sparse optical flow method and corrected. This makes it possible to continuously and stably track traffic participants in the repaired image, avoid the problem of inaccurate target association caused by drone movement, and reduce the impact of camera movement on target tracking.
[0116] In some embodiments, the affine transformation matrix includes a rotational transformation and a translational transformation;
[0117] Based on the affine transformation matrix, the coordinates of each target point corresponding to the previous frame are corrected and transformed to obtain the corrected coordinates of the target point in the current frame, including:
[0118] Based on the rotation transformation and the translation transformation, an affine transformation is performed on the coordinates of each target point corresponding to the previous frame to obtain the corrected coordinates of the target point in the current frame.
[0119] Specifically, the target point coordinates include the horizontal coordinate of the center point of the target detection frame corresponding to the target point feature, the vertical coordinate of the center point, the width of the target detection frame, and the height of the target detection frame. It should be noted that the state value corresponding to the target point feature may include, in addition to the target point coordinates, the change rate corresponding to the horizontal coordinate of the center point, the change rate corresponding to the vertical coordinate of the center point, the width change rate, and the height change rate.
[0120] Specifically, the affine transformation matrix can transform points or shapes in an image through a linear transformation and a translation transformation, and can maintain the straight lines and parallel relationships in the repaired image, but can obtain transformations in their size, direction or position. That is, the affine transformation matrix usually consists of two parts: rotation transformation and translation transformation. Rotation transformation and translation transformation can be used independently or in combination to achieve the desired transformation effect.
[0121] Among them, the rotation transformation is performed around a fixed point. In the embodiment of the present application, the center point of the target detection frame is selected as the fixed point. In two-dimensional space, it can be implemented through a rotation matrix. The rotation matrix can be defined according to the rotation angle. When the rotation matrix is applied to the target point coordinates, the point will be rotated around the rotation center by a specified angle.
[0122] In addition, in the target detection algorithm, in order to predict the target point coordinates of the current frame, the coordinate information of the target detection frame in the previous frame and the affine transformation matrix can be used to obtain them. Of course, in order to predict the target point coordinates of the next frame, the coordinate information of the target detection frame in the current frame and the affine transformation matrix can be used to obtain them. Specifically, when the target point coordinates of the current frame are obtained by the correction transformation, the initial tensor of the target detection frame of the previous frame can be determined first. The initial tensor includes the horizontal coordinate of the center point of the target detection frame, the vertical coordinate of the center point, the width of the target detection frame, the height of the target detection frame, the change rate corresponding to the horizontal coordinate of the center point, the change rate corresponding to the vertical coordinate of the center point, the width change rate and the height change rate.
[0123] Among them, the initial tensor can be expressed as: x k =[x,y,r,h,v x ,v y ,v r ,v h ], where x k represents the initial tensor, x represents the horizontal coordinate of the center point of the target detection box, y represents the vertical coordinate of the center point, r represents the width of the target detection box, h represents the height of the target detection box, and v x Indicates the rate of change of the horizontal coordinate of the center point, v yIndicates the rate of change of the vertical coordinate of the center point, v r represents the width change rate, v h Indicates the rate of change of altitude.
[0124] In addition, the initial tensor can be transformed by an affine transformation matrix including a rotation transformation and a translation transformation, wherein the rotation transformation part is used to adjust the direction of the initial tensor according to the predicted angle in the affine transformation matrix, and the translation transformation part is used to adjust the position of the initial tensor according to the predicted displacement change. The initial tensor is affine transformed according to the rotation transformation and the translation transformation to obtain the transformed predicted tensor, which can be expressed by the following formula:
[0125] in, Represents the predicted tensor, R represents the rotation transformation, x k represents the initial tensor, and T represents the translation transformation.
[0126] Furthermore, the method for determining the rotation transformation and the translation transformation can be: for the extracted image features, a number of target point features are randomly sampled, the target point features are tracked in each subsequent frame of the video image information through the sparse optical flow algorithm, and the affine transformation matrix is obtained by solving the offset of multiple target points in the previous frame and the current frame in adjacent frames through the random sample consensus (RANSAC) algorithm. The affine transformation matrix can be expressed by the following formula:
[0127]
[0128] in, represents the affine transformation matrix between the previous frame and the current frame, f-1 represents the previous frame in the adjacent frames, and f represents the current frame;
[0129] Represents the rotation transformation, which is used to represent the rotation change process of each target point coordinate between the previous frame and the current frame in adjacent frames. It is obtained by a series of trigonometric functions. 00 , R 01 , R 10 , R 11 Respectively represent the rotation parameters. For example, if the current frame and the current frame are rotated counterclockwise, then Similarly, if the current frame and the current frame are rotated clockwise, then Where θ represents the rotation angle;
[0130] represents the translation transformation, T x Represents the horizontal coordinate transformation value of the translation transformation, T y Represents the vertical coordinate transformation value of the translation transformation.
[0131] In addition, the target detection frame of the current frame can be predicted by the prediction tensor, that is, the target detection frame of the current frame can be predicted by the horizontal coordinate of the center point, the vertical coordinate of the center point, the width of the transformed target detection frame, and the height of the transformed target detection frame in the prediction tensor. The above process can be called predicting the position of the target detection frame in the current frame through Kalman filtering.
[0132] Furthermore, the intersection-and-union ratio between the detected target detection frame of the current frame and the predicted target detection frame of the current frame is determined, so as to realize the tracking of target information, that is, the information of the distance matrix corresponding to the intersection-and-union ratio and the cosine distance matrix can be merged to realize the tracking of target information.
[0133] According to the technical solution provided in the embodiment of the present application, when the specific motion trajectory of the traffic participant is uncertain, the possible position of the traffic participant in the current frame or future frame can be predicted only by the information of the previous frame and the affine transformation matrix, thereby increasing the robustness and stability of the target tracking algorithm.
[0134] In some embodiments, extracting image features from the original image by using a preset convolutional neural network, and repairing the original image based on the image features to obtain a repaired image, includes:
[0135] For each frame of the original image, based on the image features and the pixel values corresponding to the previous frame of the original image, the mask attention value corresponding to the current frame is obtained;
[0136] Based on the mask attention value corresponding to the current frame and the pixel value corresponding to the original image of the previous frame, update the preset mask matrix;
[0137] The coverage area of the updated preset mask matrix on each frame of the original image is determined, and the coverage area is repaired to obtain a repaired image.
[0138] Specifically, since the original image acquired by the shooting device is updated and transmitted in the form of a video stream, there is a temporal relationship between adjacent frames of the original image, and there is also an internal spatial information association within a single frame. In addition, the proportion of blur and afterimage (i.e., the area to be repaired) in the original image is relatively small. Therefore, the embodiment of the present application uses a mask to limit the area to be repaired to a local area to obtain a more accurate repaired image.
[0139] In addition, the preset convolutional neural network model is used to extract features of the input original image to obtain image features, and the preset mask and the image features corresponding to each frame of the original image are input into the preset mask attention network model, and a set of empty query vectors are initialized to obtain the mask attention value, where the mask attention value can reflect the importance and dependency of different pixels or areas to be repaired in the repair process, and the mask attention value can be determined by the following formula:
[0140]
[0141] Among them, X i represents the mask attention value corresponding to the original image of the current frame, and softmax represents the mask attention function. represents the pixel value corresponding to the original image of the previous frame, W represents the query vector, K represents the key vector, and K T represents the transpose of the key vector, V represents the value vector, and X i-1 Represents the mask attention value corresponding to the original image of the previous frame.
[0142] The pixel value can also be understood as the value corresponding to the preset mask matrix. The preset mask matrix is a matrix initialized to all 1s at the beginning of image restoration. It is updated after the mask attention value of each frame is updated. The pixel value corresponding to the preset mask matrix It can be expressed by the following formula:
[0143]
[0144] in, Indicates the pixel value corresponding to the pixel point with abscissa x and ordinate y. Indicates the pixel value corresponding to the horizontal coordinate x and the vertical coordinate y of the previous frame.
[0145] In addition, the updated query vector and the preset mask matrix may be passed to the next frame.
[0146] In addition, according to the updated preset mask matrix, the specific area to be repaired on each frame of the original image is determined, and the area to be repaired is determined as the covered area. The covered area can be repaired by the diffusion model to remove noise and afterimages until all the covered areas are repaired, and the repaired area is merged with the undamaged part of the original image to obtain a repaired image.
[0147] In addition, the repair method can also be an interpolation algorithm, a patch-based algorithm, a deep learning method, etc.
[0148] According to the technical solution provided in the embodiment of the present application, by extracting image features through a preset convolutional neural network and combining a mask attention mechanism and a dynamic repair strategy, the original image can be efficiently and accurately repaired, so that the repaired image is visually more complete and coherent, and the damaged or missing parts in the original image are effectively reduced or eliminated.
[0149] Figure 2 is a flowchart of another multi-task joint execution method provided in an embodiment of the present application, such as Figure 2 As shown, image information, i.e., the original images of multiple consecutive frames, is obtained, and then the image features of each frame of the original image are extracted through a preset convolutional neural network. The original image is repaired according to the image features to obtain a repaired image, and road features are extracted based on the repaired image. Road semantic segmentation is performed according to each repaired image, and the corresponding road topology map is iteratively expanded according to the segmentation result. At the same time, the repaired image is subjected to a target detection algorithm through the image features of each frame to obtain the target detection frame corresponding to the traffic participants in the repaired image of each frame, and motion compensation is performed on the target detection frame of each traffic participant to track the target, wherein motion compensation is to correct the deviation of the background motion caused by the rigid motion of the camera of the shooting device through affine transformation, and the coordinates of the corrected target detection frame are determined by Kalman filtering to predict the position of the target detection frame, and the target object is tracked by the intersection and union between the predicted target detection frame and the current target detection frame.
[0150] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0151] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.
[0152] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0153] Figure 3 Schematic diagram of a multi-task joint execution device provided in an embodiment of the present application. Figure 3 As shown, the multi-task joint execution device includes: an acquisition module 301, a repair module 302, a road processing module 303, and a tracking module 304, wherein:
[0154] An acquisition module 301 is configured to acquire original images of a plurality of adjacent frames;
[0155] The restoration module 302 is configured to extract image features from the original image through a preset convolutional neural network, and restore the original image based on the image features to obtain a restored image;
[0156] The road processing module 303 is configured to extract road features from each frame of the restored image, associate the road features corresponding to adjacent frames, obtain the association relationship between the road features of adjacent frames, and construct a road topology map in real time based on the association relationship;
[0157] The tracking module 304 is configured to modify the target detection frame corresponding to the traffic participant in the repaired image based on the image features to track the traffic participant.
[0158] In some embodiments, the road processing module 303 is configured to: perform binary classification on each frame of the repaired image through a preset image segmentation model to obtain a binary classification image; the binary classification image is used to segment the road and the background other than the road; the road features corresponding to each frame are extracted from the binary classification image corresponding to each frame, and the road features include key point features and line features corresponding to the lines between the key points; determine the correlation between the road features of the previous frame and the road features of the current frame in the adjacent frames, and based on the correlation, determine the phase difference between the repaired images of two adjacent frames; construct a road topology map based on the road features corresponding to the adjacent frames, the phase difference between the repaired images of two adjacent frames and the positioning parameters of the shooting device, wherein the original image is obtained by shooting the device.
[0159] In some embodiments, the road processing module 303 is configured to: determine the measurement value of the nth line feature and the measurement value of the jth key point feature in the current frame, and determine the mth line feature corresponding to the nth line feature and the i-th key point feature corresponding to the j-th key point feature in the previous frame; based on the uncertainty of line feature extraction and the measurement value of the nth line feature in the current frame, establish a first association relationship between the nth line feature of the current frame and the m-th line feature of the previous frame; based on the uncertainty of point feature extraction and the measurement value of the j-th key point feature in the current frame, establish a second association relationship between the j-th point feature of the current frame and the i-th point feature of the previous frame; based on the first association relationship, the second association relationship and the posture of the shooting device when measuring the measurement value of the nth line feature, determine the phase difference between the repaired images of adjacent frames.
[0160] In some embodiments, the road processing module 303 is configured to: determine the key point features corresponding to the previous frame as the first vertex set, determine the line features corresponding to the previous frame as the first edge set, and construct the road topology map to be updated based on the first vertex set and the first edge set; determine the key point features corresponding to the current frame as the second vertex set, determine the line features corresponding to the current frame as the second edge set, and construct a reference road topology map based on the second vertex set and the second edge set;
[0161] Based on the phase difference between the repaired images of two adjacent frames and the positioning parameters of the shooting device, the estimated azimuth and distance of the displacement of the shooting device are determined; based on the estimated azimuth and distance of the displacement of the shooting device, the road topology map to be updated and the reference road topology map are compared in the direction corresponding to the estimated azimuth, and the vertices that meet the update conditions are determined as update vertices, wherein the update conditions include being in the direction corresponding to the estimated azimuth, not being included in the first vertex set and being included in the second vertex set;
[0162] The first vertex set and the first edge set are updated based on the updated vertices, and the road topology graph to be updated is updated based on the updated first vertex set and the first edge set to obtain a road topology graph.
[0163] In some embodiments, tracking module 304 is configured to:
[0164] Based on the target tracking algorithm, the traffic participants in the repaired image of each frame are detected to obtain the target detection frame corresponding to each frame; multiple target point features are randomly sampled on the image features corresponding to each frame, and the target point features corresponding to each frame are tracked in the subsequent frames of each frame to determine the target point coordinates corresponding to each target point feature in each frame; based on the offset between the target point coordinates of the same target point feature in the previous frame and the current frame, the affine transformation matrix is determined; based on the affine transformation matrix, the coordinates of each target point corresponding to the previous frame are corrected and transformed to obtain the corrected target point coordinates of the current frame; based on the corrected target point coordinates of the current frame, the target detection frame of the current frame is predicted, and the intersection-and-union ratio between the detected target detection frame of the current frame and the predicted target detection frame of the current frame is determined; and traffic participants are tracked based on the intersection-and-union ratio.
[0165] In some embodiments, the affine transformation matrix includes a rotation transformation and a translation transformation; the tracking module 304 is configured to perform an affine transformation on the coordinates of each target point corresponding to the previous frame based on the rotation transformation and the translation transformation to obtain the corrected coordinates of the target point in the current frame.
[0166] In some embodiments, repair module 302 is configured to:
[0167] For each frame of the original image, the mask attention value corresponding to the current frame is obtained based on the image features and the pixel values corresponding to the previous frame of the original image; based on the mask attention value corresponding to the current frame and the pixel values corresponding to the previous frame of the original image, the preset mask matrix is updated; the coverage area of the updated preset mask matrix on each frame of the original image is determined, and the coverage area is repaired to obtain a repaired image.
[0168] Figure 4 Schematic diagram of an electronic device 4 provided in an embodiment of the present application. Figure 4As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 401 executes the computer program 403, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0169] The electronic device 4 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 4 may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that Figure 4 The electronic device 4 is merely an example and does not limit the electronic device 4 , and may include more or less components than those shown in the figure, or different components.
[0170] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0171] The memory 402 may be an internal storage unit of the electronic device 4, for example, a hard disk or memory of the electronic device 4. The memory 402 may also be an external storage device of the electronic device 4, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. The memory 402 may also include both an internal storage unit and an external storage device of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device.
[0172] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0173] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable storage media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0174] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for joint execution of multiple tasks, characterized in that: include: Acquire original images of multiple adjacent frames; Extracting image features from the original image by using a preset convolutional neural network, and repairing the original image based on the image features to obtain a repaired image; Extracting road features from the repaired image of each frame, correlating the road features corresponding to adjacent frames, obtaining the correlation relationship between the road features of adjacent frames, and constructing a road topology map in real time based on the correlation relationship; Based on the image features, the target detection frame corresponding to the traffic participant in the repaired image is corrected to track the traffic participant.
2. The method according to claim 1, characterized in that Extracting road features from the repaired image of each frame, correlating the road features corresponding to adjacent frames, obtaining the correlation relationship between the road features of adjacent frames, and constructing a road topology map in real time based on the correlation relationship, including: Perform binary classification on each frame of the repaired image by using a preset image segmentation model to obtain a binary classification image; the binary classification image is used to segment the road and the background other than the road; Extracting road features corresponding to each frame from the binary classification images corresponding to each frame, wherein the road features include key point features and line features corresponding to the lines connecting the key points; Determine the correlation relationship between the road features of the previous frame and the road features of the current frame in adjacent frames, and determine the phase difference between the restored images of two adjacent frames based on the correlation relationship; A road topology map is constructed based on road features corresponding to adjacent frames, a phase difference between restored images of two adjacent frames, and positioning parameters of a shooting device, wherein the original image is obtained by shooting the shooting device.
3. The method according to claim 2, characterized in that The determining of the correlation relationship between the road features of the previous frame and the road features of the current frame in the adjacent frames, and determining the phase difference between the restored images of the two adjacent frames based on the correlation relationship, includes: Determine the measurement value of the nth line feature and the measurement value of the jth key point feature in the current frame, and determine the mth line feature corresponding to the nth line feature and the ith key point feature corresponding to the jth key point feature in the previous frame; Based on the uncertainty of line feature extraction and the measurement value of the nth line feature in the current frame, a first association relationship between the nth line feature of the current frame and the mth line feature of the previous frame is established; Based on the uncertainty of point feature extraction and the measurement value of the j-th key point feature in the current frame, a second association relationship between the j-th point feature of the current frame and the i-th point feature of the previous frame is established; Based on the first association relationship, the second association relationship and the posture of the shooting device when measuring the measurement value of the nth line feature, the phase difference between adjacent frame repaired images is determined.
4. The method according to claim 2, characterized in that: The method of constructing a road topology map based on the road features corresponding to the adjacent frames, the phase difference between the restored images of two adjacent frames and the positioning parameters of the shooting device includes: Determine the key point features corresponding to the previous frame as a first vertex set, determine the line features corresponding to the previous frame as a first edge set, and construct a road topology graph to be updated based on the first vertex set and the first edge set; Determine the key point features corresponding to the current frame as a second vertex set, determine the line features corresponding to the current frame as a second edge set, and construct a reference road topology graph based on the second vertex set and the second edge set; Determine the estimated position and distance of the displacement of the photographing device based on the phase difference between the repaired images of two adjacent frames and the positioning parameters of the photographing device; Based on the estimated position and distance of the displacement of the photographing device, the road topology map to be updated is compared with the reference road topology map in the direction corresponding to the estimated position, and a vertex that meets an update condition is determined as an update vertex, wherein the update condition includes that the vertex is not included in the first vertex set and is included in the second vertex set in the direction corresponding to the estimated position; The first vertex set and the first edge set are updated based on the updated vertices, and the road topology graph to be updated is updated based on the updated first vertex set and the first edge set to obtain the road topology graph.
5. The method according to claim 1, characterized in that Based on the image features, the target detection frame corresponding to the traffic participant in the repaired image is corrected to track the traffic participant, including: Based on the target tracking algorithm, the traffic participants in the repaired image of each frame are detected to obtain the target detection frame corresponding to each frame; A plurality of target point features are randomly sampled on the image features corresponding to each frame, and the target point features corresponding to each frame are tracked in the subsequent frames of each frame to determine the target point coordinates corresponding to each target point feature in each frame; Determine an affine transformation matrix based on an offset between target point coordinates of the same target point feature in a previous frame and a current frame; Based on the affine transformation matrix, the coordinates of each target point corresponding to the previous frame are corrected and transformed to obtain the corrected coordinates of the target point in the current frame; Predicting a target detection frame of the current frame based on the corrected target point coordinates of the current frame, and determining an intersection-over-union ratio between the detected target detection frame of the current frame and the predicted target detection frame of the current frame; The traffic participant is tracked based on the intersection-and-merge ratio.
6. The method according to claim 5, characterized in that The affine transformation matrix includes a rotation transformation and a translation transformation; The method of performing a correction transformation on the coordinates of each target point corresponding to the previous frame based on the affine transformation matrix to obtain the corrected coordinates of the target point in the current frame includes: Based on the rotation transformation and the translation transformation, an affine transformation is performed on the coordinates of each target point corresponding to the previous frame to obtain the corrected coordinates of the target point in the current frame.
7. The method according to claim 1, characterized in that Extracting image features from the original image by using a preset convolutional neural network, and repairing the original image based on the image features to obtain a repaired image, including: For each frame of the original image, based on the image features and the pixel values corresponding to the previous frame of the original image, obtain a mask attention value corresponding to the current frame; Based on the mask attention value corresponding to the current frame and the pixel value corresponding to the original image of the previous frame, updating the preset mask matrix; Determine the coverage area of the updated preset mask matrix on each frame of the original image, and repair the coverage area to obtain a repaired image.
8. A multi-task joint execution device, characterized in that: include: An acquisition module is configured to acquire original images of a plurality of adjacent frames; A restoration module is configured to extract image features from the original image through a preset convolutional neural network, and restore the original image based on the image features to obtain a restored image; A road processing module is configured to extract road features from the repaired image of each frame, associate the road features corresponding to adjacent frames, obtain the association relationship between the road features of adjacent frames, and construct a road topology map in real time based on the association relationship; The tracking module is configured to correct the target detection frame corresponding to the traffic participant in the repaired image based on the image features to track the traffic participant.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
