Moving target detection method and device, electronic device and computer-readable storage medium

Abstract

The present application discloses a moving target detection method and device, an electronic device, and a computer-readable storage medium. The method comprises: obtaining a first pose of a target vehicle at a first time and a second pose of the target vehicle at a second time; obtaining a third pose of the target vehicle in a target pixel coordinate system based on the first pose and a transformation relationship; obtaining a fourth pose of the target vehicle in the target pixel coordinate system based on the second pose and the transformation relationship; obtaining a reference motion vector of the target vehicle from the first time to the second time based on the third pose and the fourth pose; and obtaining a target motion vector of the background based on the reference motion vector and the relative motion relationship between the target vehicle and the background.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a moving target detection method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the growing demand for intelligent vehicles, the on-board terminal of the target vehicle needs to perceive the environment around the target vehicle in real time during the movement of the target vehicle. The common perception method is based on the collected images, but since the target vehicle is in motion, the background of the target vehicle will have relative motion in the images collected at different times, which leads to low accuracy of the perception results obtained by perceiving the surrounding environment based on images.

[0003] Based on this, current methods determine the motion vector of the background in images captured at different times before perceiving the surrounding environment based on the image. This motion vector can then be used to compensate for the background motion. Perceiving the surrounding environment based on the motion-compensated image improves the accuracy of the perception results. Therefore, determining the motion vector of the target vehicle's background is of great significance. Summary of the Invention

[0004] The present application provides a moving target detection method and device, an electronic device, and a computer-readable storage medium to obtain a motion vector of a background of a target vehicle.

[0005] In a first aspect, a moving target detection method is provided. A first pose of a target vehicle at a first time and a second pose of the target vehicle at a second time are obtained, wherein the first pose and the second pose are both poses in a world coordinate system, the first time is a timestamp of a previous frame, the second time is a timestamp of a current frame, and both the previous frame and the current frame include a background of the target vehicle;

[0006] Obtaining a third pose of the target vehicle in a target pixel coordinate system according to the first pose and the transformation relationship, wherein the transformation relationship is a transformation relationship between the target pixel coordinate system and the world coordinate system, and the pixel coordinate system of the previous frame and the pixel coordinate system of the current frame are both the target pixel coordinate system;

[0007] Obtaining a fourth pose of the target vehicle in the target pixel coordinate system according to the second pose and the conversion relationship;

[0008] Obtaining a reference motion vector of the target vehicle from the first time to the second time according to the third posture and the fourth posture;

[0009] A target motion vector of the background is obtained according to the reference motion vector and the relative motion relationship between the target vehicle and the background.

[0010] In combination with any embodiment of the present application, after obtaining the target motion vector of the background, the method further includes:

[0011] performing motion compensation on the background in the current frame according to the target motion vector to obtain a coarse compensated frame;

[0012] By performing inter-frame difference between the previous frame and the coarse compensation frame, it is determined whether there is a moving target in the background.

[0013] In combination with any embodiment of the present application, performing motion compensation on the background in the current frame according to the target motion vector to obtain a coarse compensated frame includes:

[0014] Obtaining the heading angle of the target vehicle;

[0015] Obtaining an initial displacement compensation amount of the background of the current frame along the direction of the heading angle according to the heading angle and the target motion vector;

[0016] Obtaining a final displacement compensation amount of the background of the current frame according to the initial displacement compensation amount and the heading angle;

[0017] According to the final displacement compensation amount, motion compensation is performed on the background in the current frame to obtain the coarse compensation frame.

[0018] In combination with any embodiment of the present application, determining whether there is a moving target in the background by performing inter-frame difference between the previous frame and the coarse compensation frame includes:

[0019] Calculating optical flow vectors of the previous frame and the coarse compensation frame;

[0020] performing motion compensation on the coarse compensation frame according to the optical flow vector to obtain an optimized frame;

[0021] By performing inter-frame difference between the previous frame and the optimized frame, it is determined whether the moving target exists in the background.

[0022] In combination with any embodiment of the present application, determining whether the moving target exists in the background by performing inter-frame difference between the previous frame and the optimized frame includes:

[0023] Performing inter-frame difference between the previous frame and the optimized frame to obtain a grayscale difference of grayscale values ​​of pixels in the background;

[0024] Determining a pixel corresponding to a grayscale difference greater than or equal to a grayscale difference threshold as an abnormal pixel;

[0025] When the number of abnormal pixels is greater than a first number threshold, it is determined that the moving object exists in the background.

[0026] In combination with any embodiment of the present application, after determining that a pixel corresponding to a grayscale difference greater than or equal to a grayscale difference threshold is an abnormal pixel, the method further includes:

[0027] When the number of abnormal pixels is greater than or equal to a second number threshold and less than or equal to the first number threshold, performing edge detection on the abnormal region to determine edge pixels in the abnormal region, where the edge pixels are pixels belonging to an edge, and the abnormal region is a region including the abnormal pixels;

[0028] When the number of the edge pixels is greater than or equal to a third number threshold, it is determined that the moving object exists in the background.

[0029] In combination with any embodiment of the present application, after determining the edge pixels in the background, the method further includes:

[0030] When the number of the edge pixels is less than the third number threshold, it is determined that the moving object does not exist in the background.

[0031] In combination with any embodiment of the present application, performing motion compensation on the coarse compensation frame according to the optical flow vector to obtain an optimized frame includes:

[0032] performing motion compensation on the coarse compensation frame according to the optical flow vector to obtain a fine compensation frame;

[0033] Image enhancement is performed on the fine compensation frame to obtain the optimized frame.

[0034] In combination with any embodiment of the present application, after determining that a pixel corresponding to a grayscale difference greater than or equal to a grayscale difference threshold is an abnormal pixel, the method further includes:

[0035] When the number of abnormal pixels is less than a second number threshold, it is determined that the moving object does not exist in the background.

[0036] In combination with any embodiment of the present application, after determining that the moving target exists in the background, the method further includes:

[0037] Outputting warning information, wherein the warning information is used to indicate that there is a moving target around the target vehicle.

[0038] In combination with any embodiment of the present application, obtaining a first pose of the target vehicle at a first time and a second pose of the target vehicle at a second time includes:

[0039] When it is determined that the target vehicle is in a target state, a first posture of the target vehicle at a first time and a second posture of the target vehicle at a second time are obtained, where the target state includes at least one of the following: a starting state, a steering state, a parking state, and a speed less than or equal to a speed threshold.

[0040] In combination with any embodiment of the present application, obtaining a first pose of the target vehicle at a first time and a second pose of the target vehicle at a second time includes:

[0041] Calculating dead reckoning data of the target vehicle using a dead reckoning algorithm;

[0042] The first position of the target vehicle at a first time and the second position of the target vehicle at a second time are determined based on the dead reckoning data.

[0043] In conjunction with any embodiment of the present application, before determining the first posture of the target vehicle at the first time and the second posture of the target vehicle at the second time based on the dead reckoning data, the method further includes:

[0044] determining a speed of the target vehicle based on the dead reckoning data;

[0045] Determining the first position of the target vehicle at a first time and the second position of the target vehicle at a second time based on the dead reckoning data includes:

[0046] When the vehicle speed is within a preset range, the first position of the target vehicle at a first time and the second position of the target vehicle at a second time are determined according to the dead reckoning data.

[0047] In a second aspect, a moving target detection device is provided, the device comprising:

[0048] an acquisition unit, configured to acquire a first pose of a target vehicle at a first time and a second pose of the target vehicle at a second time, wherein the first pose and the second pose are both poses in a world coordinate system, the first time is a timestamp of a previous frame, the second time is a timestamp of a current frame, and both the previous frame and the current frame include a background of the target vehicle;

[0049] a first processing unit, configured to obtain a third pose of the target vehicle in a target pixel coordinate system based on the first pose and a transformation relationship, wherein the transformation relationship is a transformation relationship between the target pixel coordinate system and the world coordinate system, and the pixel coordinate system of the previous frame and the pixel coordinate system of the current frame are both the target pixel coordinate system;

[0050] A second processing unit is configured to obtain a fourth posture of the target vehicle in the target pixel coordinate system according to the second posture and the conversion relationship;

[0051] a third processing unit, configured to obtain a reference motion vector of the target vehicle from the first time to the second time according to the third posture and the fourth posture;

[0052] The third processing unit is further configured to obtain a target motion vector of the background according to the reference motion vector and a relative motion relationship between the target vehicle and the background.

[0053] In combination with any embodiment of the present application, the third processing unit is further configured to:

[0054] performing motion compensation on the background in the current frame according to the target motion vector to obtain a coarse compensated frame;

[0055] By performing inter-frame difference between the previous frame and the coarse compensation frame, it is determined whether there is a moving target in the background.

[0056] In combination with any embodiment of the present application, the third processing unit is configured to:

[0057] Obtaining the heading angle of the target vehicle;

[0058] Obtaining an initial displacement compensation amount of the background of the current frame along the direction of the heading angle according to the heading angle and the target motion vector;

[0059] Obtaining a final displacement compensation amount of the background of the current frame according to the initial displacement compensation amount and the heading angle;

[0060] According to the final displacement compensation amount, motion compensation is performed on the background in the current frame to obtain the coarse compensation frame.

[0061] In combination with any embodiment of the present application, the third processing unit is configured to:

[0062] Calculating optical flow vectors of the previous frame and the coarse compensation frame;

[0063] performing motion compensation on the coarse compensation frame according to the optical flow vector to obtain an optimized frame;

[0064] By performing inter-frame difference between the previous frame and the optimized frame, it is determined whether the moving target exists in the background.

[0065] In combination with any embodiment of the present application, the third processing unit is configured to:

[0066] Performing inter-frame difference between the previous frame and the optimized frame to obtain a grayscale difference of grayscale values ​​of pixels in the background;

[0067] Determining a pixel corresponding to a grayscale difference greater than or equal to a grayscale difference threshold as an abnormal pixel;

[0068] When the number of abnormal pixels is greater than a first number threshold, it is determined whether the moving object exists in the background.

[0069] In combination with any embodiment of the present application, the third processing unit is further configured to:

[0070] When the number of abnormal pixels is greater than or equal to a second number threshold and less than or equal to the first number threshold, performing edge detection on the abnormal region to determine edge pixels in the abnormal region, where the edge pixels are pixels belonging to an edge, and the abnormal region is a region including the abnormal pixels;

[0071] When the number of the edge pixels is greater than or equal to a third number threshold, it is determined that the moving object exists in the background.

[0072] In combination with any embodiment of the present application, the third processing unit is further configured to determine that the moving target does not exist in the background when the number of the edge pixels is less than the third number threshold.

[0073] In combination with any embodiment of the present application, the third processing unit is configured to:

[0074] performing motion compensation on the coarse compensation frame according to the optical flow vector to obtain a fine compensation frame;

[0075] Image enhancement is performed on the fine compensation frame to obtain the optimized frame.

[0076] In combination with any embodiment of the present application, the third processing unit is further configured to determine that the moving target does not exist in the background when the number of abnormal pixels is less than a second number threshold.

[0077] In combination with any embodiment of the present application, the moving target detection device further includes: an output unit, configured to output warning information, wherein the warning information is used to indicate that a moving target exists around the target vehicle.

[0078] In combination with any embodiment of the present application, the acquisition unit is used to obtain the first posture of the target vehicle at the first time and the second posture of the target vehicle at the second time when it is determined that the target vehicle is in a target state, and the target state includes at least one of the following: starting state, steering state, parking state, and speed less than or equal to a speed threshold.

[0079] In combination with any embodiment of the present application, the acquiring unit is configured to:

[0080] Calculating dead reckoning data of the target vehicle using a dead reckoning algorithm;

[0081] The first position of the target vehicle at a first time and the second position of the target vehicle at a second time are determined based on the dead reckoning data.

[0082] In combination with any embodiment of the present application, the acquiring unit is further configured to:

[0083] determining a speed of the target vehicle based on the dead reckoning data;

[0084] When the vehicle speed is within a preset range, the first position of the target vehicle at a first time and the second position of the target vehicle at a second time are determined according to the dead reckoning data.

[0085] In a third aspect, an electronic device is provided, characterized in that it includes: a processor and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation method thereof.

[0086] In a fourth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation method thereof.

[0087] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the method as described in the first aspect above and any possible implementation method thereof.

[0088] In a sixth aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is enabled to execute the method of the above-mentioned first aspect and any possible implementation thereof.

[0089] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0090] In an embodiment of the present application, when the moving target detection device obtains the first pose of the target vehicle at the first time and the second pose of the target vehicle at the second time, the third pose of the target vehicle in the target pixel coordinate system can be obtained based on the first pose and the transformation relationship, and the fourth pose of the target vehicle in the target pixel coordinate system can be obtained based on the second pose and the transformation relationship. Based on the third pose and the fourth pose, the reference motion vector of the target vehicle in the previous frame and the reference motion vector of the target vehicle in the current frame can be obtained. Since there is relative motion between the target vehicle and the background of the target vehicle, the moving target detection device can obtain the target motion vector of the background in the previous frame and the background in the current frame based on the reference motion vector. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0092] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0093] Figure 1 A schematic diagram of a flow chart of a moving target detection method provided in an embodiment of the present application;

[0094] Figure 2 A schematic diagram of the structure of an on-vehicle processor provided in an embodiment of the present application;

[0095] Figure 3 A schematic diagram of a flow chart of another moving target detection method provided in an embodiment of the present application;

[0096] Figure 4 A schematic structural diagram of a moving target detection device provided in an embodiment of the present application;

[0097] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0099] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0100] It should be understood that, in this application, "at least one (item)" means one or more, "more than one" means two or more, "at least two (items)" means two or three or more, and "and / or" is used to describe the relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist, where A and B can be singular or plural. The character " / " can indicate that the associated objects are in an "or" relationship, referring to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or plural. The character " / " can also represent the division sign in mathematical operations, for example, a / b = a divided by b; 6 / 3 = 2. "At least one of the following" or similar expressions.

[0101] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0102] The embodiments of this application are implemented by a moving target detection device, which can be any electronic device capable of implementing the technical solutions disclosed in the method embodiments of this application. Optionally, the moving target detection device can be one of the following: a mobile phone, a computer, a tablet computer, or an in-vehicle terminal.

[0103] It should be understood that the method embodiment of the present application can also be implemented by a processor executing computer program code. The following describes the embodiment of the present application in conjunction with the drawings in the embodiment of the present application. Figure 1 , Figure 1 It is a flow chart of a moving target detection method provided in an embodiment of the present application.

[0104] 101. Obtain a first pose of a target vehicle at a first time and a second pose of the target vehicle at a second time.

[0105] In the embodiment of the present application, the first pose and the second pose are both poses of the target vehicle in the world coordinate system, wherein the first pose includes the first position of the target vehicle at the first time and the first pose of the target vehicle at the first time, and the second pose includes the second position of the target vehicle at the second time and the second pose of the target vehicle at the second time, the first position and the second position are both positions of the target vehicle in the world coordinate system, and the first pose and the second pose are both poses of the target vehicle in the world coordinate system.

[0106] For example, the first time is 16:04:43 on November 12, 2022, the first position is 22 degrees 38 minutes north latitude and 114 degrees 5 minutes east longitude, and the first posture includes the heading angle of the target vehicle being 30 degrees 20 minutes east of north. Then, according to the first posture, it can be determined that the coordinates of the target vehicle in the world coordinate system at 16:04:43 on November 12, 2022 are 22 degrees 38 minutes north latitude and 114 degrees 5 minutes east longitude, and the heading angle of the target vehicle at 16:04:43 on November 12, 2022 is 30 degrees 20 minutes east of north.

[0107] In this embodiment of the present application, the first time is the timestamp of the previous frame, and the second time is the timestamp of the current frame. That is, the acquisition time of the previous frame is the first time, and the acquisition time of the current frame is the second time. The first time is earlier than the second time, that is, the acquisition time of the previous frame is earlier than the acquisition time of the current frame. Both the previous frame and the current frame include the background of the target vehicle, where the background of the target vehicle refers to the background of the target vehicle while it is driving.

[0108] Optionally, the previous frame and the current frame are two adjacent frames in an image sequence of the target vehicle's background. For example, the image sequence of the target vehicle's background includes image a, image b, and image c, where the timestamp of image a is earlier than the timestamp of image b, and the timestamp of image b is later than the timestamp of image c. In this case, the previous frame can be image a and the current frame can be image b, or the previous frame can be image b and the current frame can be image c.

[0109] Optionally, the moving target detection device obtains the previous frame and the current frame through an around view monitor (AVM) system, wherein the AVM is mounted on the target vehicle.

[0110] In one implementation of obtaining the first pose, the moving target detection device calculates the first pose through a dead reckoning (DR) algorithm.

[0111] In another implementation method of obtaining the first pose, the moving target detection device obtains the first pose by receiving the first pose sent by the terminal, wherein the terminal includes: a mobile phone, a computer, a tablet computer, a vehicle-mounted terminal, and a smart wearable device.

[0112] In one implementation of obtaining the second posture, the moving target detection device obtains the second posture through DR calculation.

[0113] In another implementation of acquiring the second posture, the moving target detection apparatus acquires the second posture by receiving the second posture sent by the terminal.

[0114] It should be understood that in the embodiment of the present application, the moving target detection device may execute the step of acquiring the first pose and the step of acquiring the second pose separately or simultaneously, and the present application does not limit this.

[0115] 102. Based on the first pose and the transformation relationship, a third pose of the target vehicle in the target pixel coordinate system is obtained.

[0116] In an embodiment of the present application, the conversion relationship is the conversion relationship between the target pixel coordinate system and the world coordinate system, wherein the pixel coordinate system of the previous frame and the pixel coordinate system of the current frame are both the target pixel coordinate system, that is, the coordinates of any pixel in the previous frame in the world coordinate system can be determined according to the conversion relationship, and the coordinates of any pixel in the current frame in the world coordinate system can also be determined according to the conversion relationship. Conversely, the coordinates of any point in the world coordinate system in the target pixel coordinate system can also be determined according to the conversion relationship. Optionally, the conversion relationship is the homography matrix between the target pixel coordinate system and the world coordinate system.

[0117] The moving target detection device converts the first pose into a pose in the target pixel coordinate system according to the conversion relationship, and obtains the pose of the target vehicle in the target pixel coordinate system at the first time, which is the third pose.

[0118] 103. According to the second posture and the conversion relationship, a fourth posture of the target vehicle in the target pixel coordinate system is obtained.

[0119] The moving target detection device converts the second posture into a posture in the target pixel coordinate system according to the conversion relationship, and obtains the posture of the target vehicle in the target pixel coordinate system at the second time, which is the fourth posture.

[0120] 104. Based on the third posture and the fourth posture, obtain a reference motion vector of the target vehicle from the first time to the second time.

[0121] In the embodiment of the present application, the reference motion vector is the motion vector of the target vehicle from the first time to the second time, and the reference vector may be the motion vector of the target vehicle in the previous frame and the target vehicle in the current frame.

[0122] The first pose and the second pose are both poses in the world coordinate system. The first pose is the pose of the target vehicle at the first time, and the second pose is the pose of the target vehicle at the second time. The first time is the time when the previous frame was acquired, and the second time is the time when the current frame was acquired. Therefore, the first pose represents the pose of the target vehicle in the world coordinate system when the previous frame was acquired, the second pose represents the pose of the target vehicle in the world coordinate system when the current frame was acquired, and the second pose represents the pose of the target vehicle in the world coordinate system when the current frame was acquired.

[0123] Therefore, the moving target detection device converts the first pose according to the conversion relationship to obtain the pose of the target vehicle in the pixel coordinate system of the previous frame (i.e., the target pixel coordinate system), i.e., the third pose. The moving target detection device converts the second pose according to the conversion relationship to obtain the pose of the target vehicle in the pixel coordinate system of the current frame (i.e., the target pixel coordinate system), i.e., the fourth pose. In this way, based on the third pose and the fourth pose, the motion vectors of the target vehicle in the previous frame and the target vehicle in the current frame in the target pixel coordinate system can be determined, which is the reference motion vector.

[0124] 105. Obtain a target motion vector of the background based on the reference motion vector and the relative motion relationship between the target vehicle and the background.

[0125] In the embodiments of the present application, the relative motion relationship between the target vehicle and the background represents the relative motion between the target vehicle and the background. Specifically, in the real world, when the target vehicle is in motion, there is relative motion between the target vehicle and the background of the target vehicle. Therefore, the relative motion relationship between the target vehicle at a first time and the target vehicle at a second time is the relative motion relationship between the background in the previous frame and the background in the current frame. In other words, the moving target detection device can determine the relative motion relationship between the background in the previous frame and the background in the current frame by determining the relative motion relationship between the target vehicle at a first time and the target vehicle at a second time.

[0126] During the target vehicle's motion, there is relative motion between the target vehicle and the background, meaning the vehicle's motion vector is the same as the background's. Therefore, based on the reference motion vector and the relative motion relationship, the motion vectors of the background in the previous frame and the background in the current frame in the target pixel coordinate system can be determined, which is the target motion vector.

[0127] In one possible implementation, the moving target detection device uses the negative vector of the reference motion vector as the target motion vector, that is, the direction of the reference motion vector is opposite to that of the target motion vector.

[0128] In an embodiment of the present application, a moving target detection device, after obtaining a first pose of a target vehicle at a first time and a second pose of the target vehicle at a second time, can obtain a third pose of the target vehicle in a target pixel coordinate system based on the first pose and a transformation relationship, and a fourth pose of the target vehicle in the target pixel coordinate system based on the second pose and the transformation relationship. Based on the third pose and the fourth pose, a reference motion vector for the target vehicle in the previous frame and the target vehicle in the current frame can be obtained. Because there is relative motion between the target vehicle and its background, the moving target detection device can obtain target motion vectors for the background in the previous frame and the background in the current frame based on the reference motion vector. Based on the technical solutions provided in steps 101 to 104, an embodiment of the present application also provides a possible application scenario. With the growing demand for intelligent vehicles, the onboard terminal of the target vehicle needs to perceive the environment surrounding the target vehicle in real time while the target vehicle is in motion. A common perception method is to achieve this through images captured by AVM. However, because the target vehicle is in motion, the background of the target vehicle will experience relative motion in images captured at different times, resulting in low accuracy in the perception results of the surrounding environment based on images captured by AVM.

[0129] Based on this, the current method determines the target motion vector of the background in the images captured by AVM at different times before perceiving the surrounding environment based on the images captured by AVM, and then performs motion compensation on the background based on the target motion vector, and then perceives the surrounding environment based on the motion-compensated image, which can improve the accuracy of the perception results.

[0130] The target method generally uses the optical flow method to determine the target motion vector of the background in the images collected by the AVM at different times. However, when the target vehicle is moving at a high speed (it should be understood that when the speed of the target vehicle exceeds the fast threshold, the target vehicle is moving at a fast speed. For example, if the fast threshold is 10 kilometers per hour, then a speed exceeding 10 kilometers per hour is considered fast), the target vehicle moves a large distance within the time interval between two frames of images collected by the AVM, which in turn results in a large difference in the background between the two adjacent frames of images collected by the AVM. This results in a large amount of data processing required to determine the target motion vector of the background using the optical flow method, and results in a low accuracy of the determined target motion vector of the background. In contrast, when the target vehicle is moving at a high speed, determining the target motion vector of the background based on the technical solution of steps 101 to 104 can reduce the amount of data processing and improve the accuracy of the target motion vector of the background.

[0131] As an optional implementation manner, after obtaining the target motion vector of the background, the moving target detection device further performs the following steps:

[0132] 201. Perform motion compensation on the background in the current frame according to the target motion vector to obtain a coarse compensated frame.

[0133] By executing step 201, the moving target device can align the background in the current frame with the background in the previous frame. That is, the coarse compensation frame is the image frame obtained by aligning the background in the current frame with the background in the previous frame. In one possible implementation, the moving target detection device moves the background in the current frame based on the target motion vector so that the background in the current frame is aligned with the background in the previous frame, thereby obtaining the coarse compensation frame.

[0134] 202. Determine whether the background includes a moving target by performing inter-frame difference between the previous frame and the coarse compensation frame.

[0135] In an embodiment of the present application, a moving target is a target in motion, for example, a motor vehicle in motion, another example, a non-motor vehicle in motion, and another example, a pedestrian in motion.

[0136] The moving target detection device can determine the difference between the background in the previous frame and the background in the coarse compensation frame by performing inter-frame difference between the previous frame and the coarse compensation frame, and then determine the presence of a moving target in the background or the presence of a moving target based on the difference.

[0137] In this embodiment, the moving target detection device first performs motion compensation on the background in the current frame based on the target motion vector, thereby reducing the displacement difference between the current background and the previous frame, resulting in a coarsely compensated frame. The presence of a moving target in the background is then determined by performing inter-frame difference between the previous frame and the coarsely compensated frame, thereby improving the accuracy of detecting moving targets in the background.

[0138] Based on this implementation, the present application also provides a possible application scenario. Based on this implementation, the target vehicle's onboard terminal determines the presence of a moving target in the target vehicle's background and then uses the onboard terminal to alert the vehicle's occupants of the moving target, thereby reducing safety hazards.

[0139] It should be understood that the target motion vector of the background obtained by executing steps 101 to 104 represents the target motion vector of the background of the previous frame and the background of the current frame in the target pixel coordinate system, when there is no moving target in the background. Therefore, in step 201, motion compensation is performed on the background in the current frame based on the target motion vector of the background, which can reduce background changes caused by the movement of the target vehicle. Therefore, by performing inter-frame difference between the coarse compensation frame obtained by motion compensation and the previous frame, the presence of a moving target in the background can be determined, which can reduce the impact of the target vehicle's motion on moving target detection, thereby improving the accuracy of moving target detection.

[0140] Specifically, if the current frame is not motion compensated, and the inter-frame difference between the current frame and the previous frame is performed to determine whether there is a moving target in the background, then even if there is no moving target in the background, the movement of the target vehicle may cause relative motion between the background in the previous frame and the background in the current frame, resulting in the result of the inter-frame difference being that the background in the current frame is different from the background in the previous frame, thereby causing false detection.

[0141] When motion compensation is performed on the current frame to obtain a coarse compensation frame, if there is no moving target in the background, then by performing inter-frame difference between the coarse compensation frame and the previous frame, it can be determined that the background in the coarse compensation frame is the same as that in the previous frame. Conversely, if there is a moving target in the background, then by performing inter-frame difference between the coarse compensation frame and the previous frame, it can be determined that the background in the coarse compensation frame is different from that in the previous frame. This reduces the probability of false detection and thus improves the accuracy of moving target detection.

[0142] As an optional implementation manner, the moving target detection device performs the following steps during the execution of step 201:

[0143] 301. Obtain the heading angle of the target vehicle.

[0144] In the embodiment of the present application, the heading angle of the target vehicle is the angle between the target vehicle's direction of travel and the Earth's North Pole. Optionally, the world coordinate system is a northeast celestial coordinate system, and the heading angle of the target vehicle is the angle between the target vehicle's direction of travel and the positive direction of the longitudinal axis of the world coordinate system.

[0145] In one possible implementation, the moving target detection device obtains the heading angle of the target vehicle through DR.

[0146] In another possible implementation manner, the moving target detection device receives the heading angle of the target vehicle sent by the terminal.

[0147] 302. Obtain an initial displacement compensation amount of the background of the current frame along the direction of the heading angle according to the heading angle and the target motion vector.

[0148] In one possible implementation, the moving target detection device first determines a displacement vector and an angle vector based on the target motion vector, wherein the displacement vector includes a horizontal displacement vector and a vertical displacement vector, and Δs represents the displacement vector, Δx represents the horizontal displacement vector, and Δy represents the vertical displacement vector. represents the angle vector, then Δs, Δx, Δy, Satisfy the following formula:

[0149]

[0150] Among them, S tx is the horizontal coordinate of the background in the pixel coordinate system of the previous frame, S (t-1)x is the horizontal coordinate of the background in the pixel coordinate system of the current frame, S ty is the vertical coordinate of the background in the pixel coordinate system of the previous frame, S (t-1)y is the vertical coordinate of the background in the pixel coordinate system of the current frame, is the heading angle of the target vehicle in the pixel coordinate system of the previous frame, is the heading angle of the background in the pixel coordinate system of the current frame.

[0151] If the displacement vector of the background of the current frame along the heading angle direction includes the horizontal displacement vector along the horizontal axis and the vertical displacement vector along the vertical axis, and the displacement vector is converted into the horizontal displacement vector along the vertical axis by using Trans x Represents the lateral displacement vector, using Trans y Represents the longitudinal displacement vector, then Trans x 、Trans y Satisfy the following formula:

[0152]

[0153] Then, Trans x and Trans y Convert to the pixel coordinate system of the previous frame:

[0154]

[0155] If the initial displacement compensation amount of the background of the current frame along the heading angle direction includes the initial horizontal compensation amount along the horizontal axis direction and the initial vertical compensation amount along the vertical axis direction, where S x is the initial lateral compensation amount, S y is the initial longitudinal compensation amount.

[0156] Optionally, the moving target detection device determines the angle compensation amount of the current frame by the following formula:

[0157]

[0158] in, The angle compensation amount for the background of the current frame.

[0159] 303. Obtain a final displacement compensation amount of the background of the current frame according to the initial displacement compensation amount and the heading angle.

[0160] In a possible implementation, the final displacement compensation amount of the background of the current frame includes the final horizontal compensation amount along the horizontal axis and the final vertical compensation amount along the vertical axis, and the final compensation amount is expressed as x Indicates the final lateral compensation amount, using final y Indicates the final longitudinal compensation amount, then final x 、final y Satisfy the following formula:

[0161]

[0162] 304. Perform motion compensation on the background in the current frame according to the final displacement compensation amount to obtain the coarse compensation frame.

[0163] In this embodiment, the moving target detection device determines an initial displacement compensation amount for the background of the current frame along the heading angle direction based on the heading angle and the target motion vector. The final displacement compensation amount for the background of the current frame is then determined based on the initial displacement compensation amount and the heading angle. When determining the displacement compensation amount for the background of the current frame based on the target motion vector, the heading angle of the target vehicle is taken into account, thereby improving the accuracy of the displacement compensation amount for the background of the current frame. Thus, based on the final displacement compensation amount, motion compensation is performed on the background of the current frame to produce a coarsely compensated frame, which improves the alignment of the background in the coarsely compensated frame with the background in the previous frame.

[0164] As an optional implementation manner, the moving target detection device performs the following steps during the execution of step 202:

[0165] 401. Calculate the optical flow vectors of the previous frame and the coarse compensation frame.

[0166] In one possible implementation, the moving target detection device calculates the optical flow vectors of the previous frame and the coarse compensation frame using an optical flow algorithm. Optionally, the implementation process of this implementation is as follows:

[0167] Assume that any pixel in the previous frame is represented as (x, y, t), where x is the horizontal coordinate in the pixel coordinate system of the previous frame, y is the horizontal coordinate in the pixel coordinate system of the previous frame, and t is time. The pixel brightness of the pixel (x, y, t) is represented as I(x, y, t), and the pixel corresponding to the pixel in the current frame (it should be understood that the correspondence here represents that the object point corresponding to the two pixels is the same) is represented as (x+σx, y+σy, t+σt), where σx is the displacement of the two pixels in the horizontal axis direction, σy is the displacement of the two pixels in the vertical axis direction, and σt is the acquisition time interval between the previous frame and the current frame, that is, the difference between the second time and the first time. Based on the assumption that the pixel brightness of the two pixels is the same, the following formula is obtained:

[0168] I(x,y,t)=(x+Δx,y+Δy,t+Δt)…Formula (6)

[0169] After motion compensation in step 304, the displacement difference between the background in the previous frame and the background in the current frame should be close to 0. Based on the assumption of uniform brightness, the partial derivatives of brightness in all directions can be approximated to be 0, which yields the following equation:

[0170]

[0171] Formula (7) can be transformed into the following formula:

[0172]

[0173] Then transform formula (8) into the following formula:

[0174]

[0175] The Lucas-Kanad method (also called the Lucas-Kanad method) is used to solve formula (9) to obtain the optical flow vectors of the previous frame and the coarse compensation frame.

[0176] 402. Perform motion compensation on the coarse compensation frame according to the optical flow vector to obtain an optimized frame.

[0177] By executing step 402, the moving target detection device aligns the background in the coarse compensated frame with the background in the previous frame. Specifically, the optimized frame is the image frame obtained by aligning the background in the coarse compensated frame with the background in the previous frame. In one possible implementation, the moving target detection device moves the background in the coarse compensated frame based on the optical flow vector so that the background in the coarse compensated frame is aligned with the background in the previous frame, thereby obtaining the optimized frame.

[0178] 403. Determine whether the moving target exists in the background by performing inter-frame difference between the previous frame and the optimized frame.

[0179] In this embodiment, the moving target detection device, after compensating the background in the current frame based on the target background motion vector to obtain a coarse compensated frame, calculates the optical flow vectors of the coarse compensated frame and the previous frame. This reduces the data processing required to calculate the optical flow vectors and improves the accuracy of the optical flow vectors. Motion compensation is then performed on the coarse compensated frame based on the optical flow vectors, further reducing the displacement difference between the background in the coarse compensated frame and the background in the previous frame, resulting in an optimized frame.

[0180] It should be understood that, as previously mentioned, when the target vehicle is moving quickly, the accuracy of the target motion vector of the background determined using the optical flow method is low, and the data processing required to determine the target motion vector is high. In the embodiment of the present application, the moving target detection device first determines the target motion vector of the background using the technical solution provided in steps 101 to 104. Then, in step 201, motion compensation is performed on the background in the current frame based on the target motion vector, thereby reducing the difference between the background in the current frame and the background in the previous frame, resulting in a coarsely compensated frame. At this point, the optical flow vectors of the background in the coarsely compensated frame and the background in the previous frame are calculated, and motion compensation is performed on the coarsely compensated frame based on the optical flow vectors. This reduces the data processing required and reduces motion compensation errors. That is, the embodiment of the present application first performs coarse motion compensation on the current frame through steps 101 to 104 and step 201 to obtain a coarsely compensated frame, and then performs fine motion compensation on the coarsely compensated frame through steps 401 to 403 to obtain optimized features. This reduces the data processing required, increases processing speed, and enhances the effectiveness of motion compensation.

[0181] As an optional implementation manner, the moving target detection device performs the following steps during the execution of step 403:

[0182] 501. Perform inter-frame difference between the previous frame and the optimized frame to obtain a grayscale difference of the grayscale values ​​of the pixels in the background.

[0183] For ease of description, two pixels located at the same position in the previous frame and the optimized frame are referred to as a co-located pixel pair. For example, if pixel a is located at (2, 4) in the previous frame and pixel b is located at (2, 4) in the optimized frame, then pixels a and b are a co-located pixel pair.

[0184] In the embodiment of the present application, the grayscale difference is the difference in grayscale values ​​between two pixels in a co-located pixel pair. For example, if pixel a in the previous frame and pixel b in the optimized frame are a co-located pixel pair, then the difference between the grayscale value of pixel a and the grayscale value of pixel b is the grayscale difference. Alternatively, the grayscale difference is the absolute value of the difference in grayscale values ​​between the two pixels in the co-located pixel pair.

[0185] The moving target detection device can determine the grayscale difference between two pixels in all pairs of pixels at the same position in the previous frame and the optimized frame by performing inter-frame difference between the previous frame and the optimized frame.

[0186] 502. Determine a pixel corresponding to a grayscale difference greater than or equal to a grayscale difference threshold as an abnormal pixel.

[0187] A large grayscale difference indicates that the difference between the two pixels in the same-position pixel pair is large, which means that the probability that the same-position pixel pair corresponding to the grayscale difference is the pixel of a moving target is high. Conversely, a small grayscale difference indicates that the difference between the two pixels in the same-position pixel pair is small, which means that the probability that the same-position pixel pair corresponding to the grayscale difference is the pixel of a moving target is low.

[0188] In the embodiment of the present application, the moving target detection device determines whether the grayscale difference is large or small based on the grayscale difference threshold. Specifically, if the grayscale difference is greater than or equal to the grayscale difference threshold, it indicates a large grayscale difference, and thus the corresponding pair of pixels in the same position is likely to be pixels of a moving target. If the grayscale difference is less than the grayscale difference threshold, it indicates a small grayscale difference, and thus the corresponding pair of pixels in the same position is unlikely to be pixels of a moving target.

[0189] In the embodiment of the present application, pixels that may be moving targets are referred to as abnormal pixels. Optionally, abnormal pixels are pixels in the current frame in a co-located pixel pair that may be a moving target. For example, co-located pixel pair a includes pixel b and pixel c, where pixel b is a pixel in the previous frame and pixel c is a pixel in the current frame. If the co-located pixel pair a is a pixel that may be a moving target, pixel c is an abnormal pixel.

[0190] 503. When the number of the abnormal pixels is greater than a first number threshold, determine that the moving target exists in the background.

[0191] Obviously, a large number of abnormal pixels indicates a high probability of a moving target in the background. In the embodiment of the present application, the moving target detection device determines whether the number of abnormal pixels is sufficient based on the first number threshold. Specifically, if the number of abnormal pixels is greater than the first number threshold, it indicates that the number of abnormal pixels is large, and thus determines that a moving target is present in the background.

[0192] In this embodiment, the moving target detection device obtains the grayscale difference of the grayscale values ​​of the pixels in the background by performing inter-frame difference between the previous frame and the optimized frame, and then determines the abnormal background pixels based on the grayscale difference and the grayscale difference threshold. Finally, based on the number of abnormal pixels and the first number threshold, it can be determined that there is a moving target in the background.

[0193] As an optional implementation, after determining abnormal pixels in the background, the moving target detection apparatus further performs the following steps: 601. When the number of the abnormal pixels is less than a second number threshold, determine that the moving target does not exist in the background.

[0194] A small number of abnormal pixels indicates a low probability of a moving target in the background. In this embodiment of the present application, the moving target detection device determines whether the number of abnormal pixels is sufficiently small based on a second number threshold. Specifically, if the number of abnormal pixels is less than the second number threshold, it indicates that the number of abnormal pixels is small, and thus, it is determined that there is no moving target in the background.

[0195] As described in step 503 and step 601, if the number of abnormal pixels is greater than the first number threshold, it means that there are many abnormal pixels, and if the number of abnormal pixels is less than the second number threshold, it means that there are few abnormal pixels. Then, when the number of abnormal pixels is between many and few, it means that it is difficult to determine whether there is a moving target in the background based on the number of abnormal pixels.

[0196] Therefore, when the number of abnormal pixels is greater than or equal to the second number threshold and less than or equal to the first number threshold, the moving target detection device further determines whether there is a moving target in the background by performing edge detection on the background.

[0197] As an optional implementation, after determining abnormal pixels in the background, the moving target detection device further performs the following steps:

[0198] 701 : When the number of abnormal pixels is greater than or equal to the second number threshold and less than or equal to the first number threshold, perform edge detection on the abnormal region to determine edge pixels in the abnormal region.

[0199] In the embodiment of the present application, the abnormal region is a region including abnormal pixels, and the area of ​​the abnormal region is smaller than the area of ​​the background, that is, the abnormal region is a region in the background where a moving object may exist. Optionally, the abnormal region is the minimum adjacent region of the abnormal pixels, for example, the abnormal pixels are the minimum adjacency matrix of the abnormal pixels.

[0200] In the embodiment of the present application, edge pixels are pixels belonging to an edge. The moving target detection device can detect edge pixels in the abnormal region by performing edge detection on the abnormal region. Optionally, edge detection is implemented using a Canny edge detection algorithm. Specifically, the moving target detection device extracts edge features from the abnormal region using the Canny edge detection algorithm to obtain an edge gradient value, and then determines the edge pixels in the abnormal region by statistically analyzing the edge gradient value.

[0201] 702. When the number of the edge pixels is greater than or equal to a third number threshold, determine that the moving target exists in the background.

[0202] A large number of edge pixels indicates that there are obvious edges in the abnormal area, which means that the probability of a moving target existing in the abnormal area is high. Conversely, a small number of edge pixels indicates that there are no obvious edges in the abnormal area, which means that the probability of a moving target existing in the abnormal area is low.

[0203] In the embodiment of the present application, the moving target detection device determines whether the number of edge pixels is large or small based on a third number threshold. Specifically, if the number of edge pixels is greater than or equal to the third number threshold, it indicates that the number of edge pixels is large, and thus, a moving target is determined to be present in the abnormal region. Conversely, if the number of edge pixels is less than the third number threshold, it indicates that the number of edge pixels is small, and thus, no moving target is determined to be present in the abnormal region.

[0204] Therefore, the moving target detection device determines that there is a moving target in the background when the number of edge pixels is greater than or equal to the third number threshold, and determines that there is no moving target in the background when the number of edge pixels is less than the third number threshold.

[0205] In this embodiment, when it is difficult to determine whether there is a moving target in the background based on the number of abnormal pixels, the moving target detection device performs edge detection on the abnormal area to determine the edge pixels in the abnormal area, and then determines whether there is a moving target in the background based on the number of edge pixels and the third quantity threshold, thereby improving the detection accuracy of the moving target detection.

[0206] As an optional implementation manner, the moving target detection device performs the following steps during the execution of step 402:

[0207] 801. Perform motion compensation on the coarse compensation frame according to the optical flow vector to obtain a fine compensation frame.

[0208] The implementation of this step can refer to the implementation of step 402. It should be noted that in this step, motion compensation is performed on the coarse compensation frame according to the optical flow vector, and what is obtained is not an optimized frame, but a fine compensation frame.

[0209] 802. Perform image enhancement on the fine compensation frame to obtain the optimized frame.

[0210] The moving target detection device can enhance the finely compensated frame to remove noise and enhance image quality. Specifically, by performing step 802 to obtain an optimized frame, noise in the optimized frame can be reduced, thereby improving the image quality of the optimized frame. Optionally, the moving target detection device can enhance the finely compensated frame using image morphological operations to obtain the optimized frame, where the image morphological operations include erosion and dilation.

[0211] In this embodiment, after obtaining the fine compensation frame, the moving target detection device performs image enhancement on the fine compensation frame to obtain an optimized frame, which can reduce the noise of the optimized frame and improve the image quality of the optimized frame.

[0212] As an optional embodiment, after determining that there is a moving target in the background, the moving target detection device further performs the following steps: outputting an alarm message, wherein the alarm message is used to indicate the presence of a moving target around the target vehicle. In one possible implementation, the moving target detection device displays the alarm message through an onboard terminal of the target vehicle. In another possible implementation, the moving target detection device outputs the alarm message through a speaker of the target vehicle. In yet another possible implementation, the moving target detection device displays the alarm message through a head-up display installed on the target vehicle.

[0213] In this embodiment, when the moving target detection device determines that there is a moving target in the background, it outputs an alarm message to remind people in the target vehicle that there is a moving target around the target vehicle, thereby reducing safety hazards.

[0214] As an optional embodiment, the moving target detection device obtains the first posture of the target vehicle at a first time and the second posture of the target vehicle at a second time when determining that the target vehicle is in a target state, wherein the target state includes at least one of the following: starting state, steering state, parking state, and speed less than or equal to a speed threshold.

[0215] In an embodiment of the present application, the start state indicates that the target vehicle has been started. For example, the driver presses the start button of the target vehicle to change the target vehicle from an unstarted state to a started state. The turning state indicates that the target vehicle is turning. For example, the driver toggles the turn signal switch of the target vehicle to put the target vehicle into a turning state. The parking state indicates that the target vehicle is parking. For example, when the driver inputs a parking instruction to the on-board terminal of the target vehicle, the target vehicle enters a parking state. For another example, when the target vehicle is in a reversing state, the target vehicle enters a parking state. The speed threshold is the basis for determining whether the speed of the target vehicle is fast or slow. Specifically, if the speed of the target vehicle is less than or equal to the speed threshold, it means that the speed of the target vehicle is slow. If the speed of the target vehicle is greater than the speed threshold, it means that the speed of the target vehicle is fast.

[0216] In this embodiment, the moving target detection device obtains the first posture of the target vehicle at the first time and the second posture of the target vehicle at the second time when the target vehicle is in the target state, that is, the moving target detection device executes step 101 when the target vehicle is in the target state, and then performs moving target detection on the background around the target based on the first posture and second posture obtained by executing step 101, which can reduce the safety hazards of the target vehicle in the target state.

[0217] In other words, in this embodiment, the target vehicle being in the target state can be considered a condition for the moving target detection device to perform moving target detection. Alternatively, performing moving target detection on the target vehicle's background is an autonomous driving function of the target vehicle, and the target vehicle being in the target state can be considered an activation condition for the autonomous driving function.

[0218] As an optional implementation manner, the moving target detection device obtains a first pose of the target vehicle at a first time and a second pose of the target vehicle at a second time by performing the following steps:

[0219] 901. Calculate the dead reckoning data of the target vehicle through DR.

[0220] In an embodiment of the present application, the dead reckoning data is the motion data of the target vehicle calculated by DR. Optionally, the dead reckoning data includes at least one of the following: the position of the target vehicle, the heading angle of the target vehicle, the speed of the target vehicle, and the acceleration of the target vehicle.

[0221] 902. Determine the first pose of the target vehicle at the first time and the second pose of the target vehicle at the second time based on the dead reckoning data.

[0222] The dead reckoning data includes the position of the target vehicle in the world coordinate system at any time. Therefore, the moving target detection device can determine the position of the target vehicle at the first time as the first position based on the dead reckoning data, and can also determine the position of the target vehicle at the second time as the second position.

[0223] As an optional implementation manner, the moving target detection device further performs the following steps before performing step 902:

[0224] 1001. Determine the speed of the target vehicle based on the dead reckoning data.

[0225] In the case where the dead reckoning data includes the speed of the target vehicle, the moving target detection device may determine the speed of the target vehicle based on the dead reckoning data.

[0226] When the speed of the target vehicle is determined, the moving target detection device determines a first posture of the target vehicle at a first time and a second posture of the target vehicle at a second time by performing the following steps:

[0227] 1002. When the vehicle speed is within a preset range, determine the first posture of the target vehicle at the first time and the second posture of the target vehicle at the second time based on the dead reckoning data.

[0228] Because the dead reckoning data obtained from DR calculations may contain errors, when the errors in the dead reckoning data are large, the accuracy of both the first and second poses determined by the moving target detection device based on the dead reckoning data is low, which in turn leads to low detection accuracy of moving targets based on the first and second poses. Therefore, when the errors in the dead reckoning data are small, determining the first and second poses based on the dead reckoning data can improve the accuracy of the first and second poses, thereby improving the accuracy of moving target detection.

[0229] The target vehicle's speed has a reasonable range, and the target vehicle's speed is derived from dead reckoning data. If the target vehicle's speed is within this range, the error in the dead reckoning data is small. If the target vehicle's speed is outside this range, the error in the dead reckoning data is large. For example, if a vehicle's speed is typically under 140 kilometers per hour, then the reasonable range could be 0 to 140 kilometers per hour.

[0230] In the embodiment of the present application, the preset interval is the above-mentioned reasonable interval. Therefore, when the speed of the target vehicle is within the preset interval, the moving target detection device determines the first posture and the second posture of the target vehicle based on the track calculation data, which can improve the accuracy of the first posture and the accuracy of the second posture, thereby improving the accuracy of moving target detection.

[0231] Optionally, since in an embodiment of the present application, the moving target detection device needs to use track calculation data to realize moving target detection, when the speed of the target vehicle is outside the preset range, the moving target detection device determines that the error of the track calculation data is large, and then does not perform moving target detection, thereby reducing the large error in the detection result of the moving target detection due to the error of the track calculation data, thereby reducing misleading to the personnel in the target vehicle and reducing safety hazards.

[0232] Based on the technical solutions provided in the embodiments of this application, the embodiments of this application also provide a possible application scenario. By executing the technical solutions provided above, the onboard processor of the target vehicle can detect whether there are any moving targets around the target vehicle, and can then output an alarm message when there are moving targets around the target vehicle, thereby reducing safety risks.

[0233] In one possible implementation, Figure 2 The following is a schematic diagram of the structure of the on-board processor of the target vehicle, as shown in FIG. Figure 2 As shown, the on-board processor can obtain the images collected by the AVM and the dead reckoning data of the target vehicle obtained by the DR module through DR calculation. The on-board processor can also establish a communication connection with an interface processor (IP). An electronic device for burning computer program code can burn the computer program code to the on-board processor through the IP. Specifically, the electronic device for burning computer program code establishes a communication connection with the on-board processor through a controller area network (CAN) bus and the IP, and burns the computer program code to the on-board processor through the communication connection. The on-board processor can also establish a communication connection with a head unit (HUT). The on-board processor can output data to the HUT through the communication connection. For example, if the head unit includes an on-board display, the on-board processor can communicate with the on-board display through the IP.

[0234] Figure 3 The figure shows a flow chart of the onboard processor of the target vehicle detecting whether there is a moving target around the target vehicle by executing the technical solution provided above, and outputting warning information when there is a moving target around the target vehicle. Figure 3 As shown, in step 31, the onboard processor obtains the previous frame, current frame, and the aforementioned conversion relationship output by the AVM from the AVM. Optionally, the previous frame and current frame are both bird's-eye views obtained by stitching images obtained by the AVM of the surroundings of the target vehicle. In step 32, dead reckoning data is obtained from the DR module. Then, in step 33, motion compensation is performed on the background in the previous frame based on the previous frame, current frame, conversion relationship, and dead reckoning data to obtain a finely compensated frame. In step 34, the finely compensated frame and the previous frame are grayscaled, and in step 35, the finely compensated grayscale image is enhanced to obtain the aforementioned optimized frame. In step 36, inter-frame difference is performed on the grayscale images of the optimized frame and the previous frame to determine abnormal pixels.

[0235] In step 37, based on the number of abnormal pixels, it is determined whether there is a moving target in the background, and whether it is necessary to further determine whether there is a moving target through edge detection. Specifically, if the number of abnormal pixels is greater than a first number threshold, it is determined that there is a moving target in the background, and an alarm message is output in step 38. If the number of abnormal pixels is less than a second number threshold, it is determined that there is no moving target in the background in step 39. If the number of abnormal pixels is greater than or equal to the second number threshold and less than or equal to the first number threshold, edge detection is performed on the abnormal region in step 40 to determine the edge pixels in the abnormal region. In step 41, based on the number of edge pixels, it is determined whether there is a moving target in the background. Specifically, if the number of edge pixels is greater than or equal to a third number threshold, it is determined that there is a moving target in the background, and an alarm message is output in step 42. If the number of edge pixels is less than the third number threshold, it is determined that there is no moving target in the background in step 43.

[0236] Optionally, the on-board processor implements moving target detection through a moving target detection system, wherein the moving target detection system includes: a DR parameter parsing module, a coordinate system establishment and conversion module, a motion compensation parameter calculation module, a motion compensation processing module, an optical flow vector analysis module, an image precision compensation module, an inter-frame difference calculation module, a coarse detection module, a local edge feature statistics module, and a fine detection module. The DR parameter parsing module is used to determine the speed of the target vehicle based on the dead reckoning data, and to determine whether the error in the dead reckoning data is large based on the speed of the target vehicle. If the error is large, the dead reckoning data is determined to be invalid and moving target detection is not performed. If the error is small, the dead reckoning data is determined to be valid and moving target detection is performed. For the specific implementation process, see steps 1001 and 1002. The coordinate system establishment and conversion module is used to determine the above-mentioned conversion relationship, the motion compensation parameter calculation module is used to calculate the target motion vector of the background, the motion compensation processing module is used to execute the above-mentioned step 201, the optical flow vector analysis module is used to execute the above-mentioned step 401, the image precision compensation module is used to execute the above-mentioned step 402, the inter-frame difference calculation module is used to execute the above-mentioned step 501 and step 502, and the coarse detection module is used to execute the above-mentioned step 503 and step 601. The local edge feature statistics module is used to execute the above-mentioned step 701, and the fine detection module is used to execute the above-mentioned step 702.

[0237] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0238] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0239] The above describes in detail the method of the embodiment of the present application, and the following provides an apparatus of the embodiment of the present application.

[0240] See also Figure 4 , Figure 4 This is a schematic structural diagram of a moving target detection device provided in an embodiment of the present application. The moving target detection device 1 includes: an acquisition unit 11, a first processing unit 12, a second processing unit 13, and a third processing unit 14. Optionally, the moving target detection device 1 also includes an output unit 15. Specifically:

[0241] An acquisition unit 11 is configured to acquire a first pose of a target vehicle at a first time and a second pose of the target vehicle at a second time, wherein the first pose and the second pose are both poses in a world coordinate system, the first time is a timestamp of a previous frame, the second time is a timestamp of a current frame, and both the previous frame and the current frame include a background of the target vehicle;

[0242] a first processing unit 12, configured to obtain a third pose of the target vehicle in a target pixel coordinate system based on the first pose and a transformation relationship, wherein the transformation relationship is a transformation relationship between the target pixel coordinate system and the world coordinate system, and the pixel coordinate system of the previous frame and the pixel coordinate system of the current frame are both the target pixel coordinate system;

[0243] A second processing unit 13 is configured to obtain a fourth posture of the target vehicle in the target pixel coordinate system according to the second posture and the conversion relationship;

[0244] A third processing unit 14 is configured to obtain a reference motion vector of the target vehicle from the first time to the second time according to the third posture and the fourth posture;

[0245] The third processing unit 14 is further configured to obtain a target motion vector of the background according to the reference motion vector and the relative motion relationship between the target vehicle and the background.

[0246] In combination with any embodiment of the present application, the third processing unit 14 is further configured to:

[0247] performing motion compensation on the background in the current frame according to the target motion vector to obtain a coarse compensated frame;

[0248] By performing inter-frame difference between the previous frame and the coarse compensation frame, it is determined whether there is a moving target in the background.

[0249] In combination with any embodiment of the present application, the third processing unit 14 is configured to:

[0250] Obtaining the heading angle of the target vehicle;

[0251] Obtaining an initial displacement compensation amount of the background of the current frame along the direction of the heading angle according to the heading angle and the target motion vector;

[0252] Obtaining a final displacement compensation amount of the background of the current frame according to the initial displacement compensation amount and the heading angle;

[0253] According to the final displacement compensation amount, motion compensation is performed on the background in the current frame to obtain the coarse compensation frame.

[0254] In combination with any embodiment of the present application, the third processing unit 14 is configured to:

[0255] Calculating optical flow vectors of the previous frame and the coarse compensation frame;

[0256] performing motion compensation on the coarse compensation frame according to the optical flow vector to obtain an optimized frame;

[0257] By performing inter-frame difference between the previous frame and the optimized frame, it is determined whether the moving target exists in the background.

[0258] In combination with any embodiment of the present application, the third processing unit 14 is configured to:

[0259] Performing inter-frame difference between the previous frame and the optimized frame to obtain a grayscale difference of grayscale values ​​of pixels in the background;

[0260] Determining a pixel corresponding to a grayscale difference greater than or equal to a grayscale difference threshold as an abnormal pixel;

[0261] When the number of abnormal pixels is greater than a first number threshold, it is determined that the moving object exists in the background.

[0262] In combination with any embodiment of the present application, the third processing unit 14 is further configured to:

[0263] When the number of abnormal pixels is greater than or equal to a second number threshold and less than or equal to the first number threshold, performing edge detection on the abnormal region to determine edge pixels in the abnormal region, where the edge pixels are pixels belonging to an edge, and the abnormal region is a region including the abnormal pixels;

[0264] When the number of the edge pixels is greater than or equal to a third number threshold, it is determined that the moving object exists in the background.

[0265] In combination with any embodiment of the present application, the third processing unit 14 is further configured to determine that the moving target does not exist in the background when the number of the edge pixels is less than the third number threshold.

[0266] In combination with any embodiment of the present application, the third processing unit 14 is configured to:

[0267] performing motion compensation on the coarse compensation frame according to the optical flow vector to obtain a fine compensation frame;

[0268] Image enhancement is performed on the fine compensation frame to obtain the optimized frame.

[0269] In combination with any embodiment of the present application, the third processing unit 14 is further configured to determine that the moving target does not exist in the background when the number of abnormal pixels is less than a second number threshold.

[0270] In combination with any embodiment of the present application, the moving target detection device 1 further includes: an output unit 15, configured to output warning information, wherein the warning information is used to indicate that there is a moving target around the target vehicle.

[0271] In combination with any embodiment of the present application, the acquisition unit 11 is used to obtain the first posture of the target vehicle at the first time and the second posture of the target vehicle at the second time when it is determined that the target vehicle is in a target state, and the target state includes at least one of the following: starting state, steering state, parking state, and speed less than or equal to a speed threshold.

[0272] In combination with any embodiment of the present application, the acquiring unit 11 is configured to:

[0273] Calculating dead reckoning data of the target vehicle using a dead reckoning algorithm;

[0274] The first position of the target vehicle at a first time and the second position of the target vehicle at a second time are determined based on the dead reckoning data.

[0275] In combination with any embodiment of the present application, the acquiring unit 11 is further configured to:

[0276] determining a speed of the target vehicle based on the dead reckoning data;

[0277] When the vehicle speed is within a preset range, the first position of the target vehicle at a first time and the second position of the target vehicle at a second time are determined according to the dead reckoning data.

[0278] In an embodiment of the present application, when the moving target detection device obtains the first pose of the target vehicle at the first time and the second pose of the target vehicle at the second time, the third pose of the target vehicle in the target pixel coordinate system can be obtained based on the first pose and the transformation relationship, and the fourth pose of the target vehicle in the target pixel coordinate system can be obtained based on the second pose and the transformation relationship. Based on the third pose and the fourth pose, the reference motion vector of the target vehicle in the previous frame and the reference motion vector of the target vehicle in the current frame can be obtained. Since there is relative motion between the target vehicle and the background of the target vehicle, the moving target detection device can obtain the target motion vector of the background in the previous frame and the background in the current frame based on the reference motion vector.

[0279] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0280] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, the memory 22, the input device 23 and the output device 24 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, connection through various interfaces, transmission lines, buses, etc.

[0281] The processor 21 may be one or more graphics processing units (GPUs). If the processor 21 is a GPU, the GPU may be a single-core GPU or a multi-core GPU. Alternatively, the processor 21 may be a processor group consisting of multiple GPUs, with the multiple processors coupled to each other via one or more buses. Alternatively, the processor may be another type of processor, etc., which is not limited in the present embodiment.

[0282] The memory 22 can be used to store computer program instructions and various computer program codes, including program codes for executing the solution of the present application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or portable compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0283] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0284] It can be understood that in the embodiment of the present application, the memory 22 can be used not only to store relevant instructions, but also to store relevant data. For example, the memory 22 can be used to store the first posture and the second posture obtained through the input device 23, or the memory 22 can also be used to store the motion vector of the background obtained by the processor 21, etc. The embodiment of the present application does not limit the specific data stored in the memory.

[0285] It is understandable that Figure 5 Only a simplified design of an electronic device is shown. In actual applications, the electronic device may further include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the scope of protection of the present application.

[0286] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0287] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here. Those skilled in the art will also clearly understand that the descriptions of the various embodiments of this application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, reference can be made to the descriptions of other embodiments.

[0288] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0289] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0290] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0291] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0292] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by a computer program instructing related hardware to perform the processes. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A moving target detection method, characterized in that: The method comprises: Obtain a first pose of a target vehicle at a first time and a second pose of the target vehicle at a second time, where the first pose and the second pose are both poses in a world coordinate system, the first time is a timestamp of a previous frame, the second time is a timestamp of a current frame, and both the previous frame and the current frame include a background of the target vehicle; Obtaining a third pose of the target vehicle in a target pixel coordinate system according to the first pose and the transformation relationship, wherein the transformation relationship is a transformation relationship between the target pixel coordinate system and the world coordinate system, and the pixel coordinate system of the previous frame and the pixel coordinate system of the current frame are both the target pixel coordinate system; Obtaining a fourth pose of the target vehicle in the target pixel coordinate system according to the second pose and the conversion relationship; Obtaining a reference motion vector of the target vehicle from the first time to the second time according to the third posture and the fourth posture; A target motion vector of the background is obtained according to the reference motion vector and the relative motion relationship between the target vehicle and the background.

2. The method according to claim 1, characterized in that After obtaining the target motion vector of the background, the method further includes: performing motion compensation on the background in the current frame according to the target motion vector to obtain a coarse compensated frame; By performing inter-frame difference between the previous frame and the coarse compensation frame, it is determined whether there is a moving target in the background.

3. The method according to claim 2, characterized in that The step of performing motion compensation on the background in the current frame according to the target motion vector to obtain a coarse compensated frame includes: Obtaining the heading angle of the target vehicle; Obtaining an initial displacement compensation amount of the background of the current frame along the direction of the heading angle according to the heading angle and the target motion vector; Obtaining a final displacement compensation amount of the background of the current frame according to the initial displacement compensation amount and the heading angle; According to the final displacement compensation amount, motion compensation is performed on the background in the current frame to obtain the coarse compensation frame.

4. The method according to claim 2, characterized in that Determining whether there is a moving target in the background by performing inter-frame difference between the previous frame and the coarse compensation frame includes: Calculating optical flow vectors of the previous frame and the coarse compensation frame; performing motion compensation on the coarse compensation frame according to the optical flow vector to obtain a fine compensation frame; Performing image enhancement on the fine compensation frame to obtain an optimized frame; By performing inter-frame difference between the previous frame and the optimized frame, it is determined whether the moving target exists in the background.

5. The method according to claim 4, characterized in that Determining whether the moving target exists in the background by performing inter-frame difference between the previous frame and the optimized frame includes: Performing inter-frame difference between the previous frame and the optimized frame to obtain a grayscale difference of grayscale values ​​of pixels in the background; Determining a pixel corresponding to a grayscale difference greater than or equal to a grayscale difference threshold as an abnormal pixel; When the number of abnormal pixels is greater than a first number threshold, determining that the moving target exists in the background; When the number of abnormal pixels is less than a second number threshold, it is determined that the moving target does not exist in the background, and the second number threshold is less than the first number threshold.

6. The method according to claim 5, characterized in that After determining that a pixel corresponding to a grayscale difference greater than or equal to the grayscale difference threshold is an abnormal pixel, the method further includes: When the number of abnormal pixels is greater than or equal to the second number threshold and less than or equal to the first number threshold, performing edge detection on the abnormal area to determine edge pixels in the abnormal area, where the edge pixels are pixels belonging to an edge, and the abnormal area is an area including the abnormal pixels; When the number of the edge pixels is greater than or equal to a third number threshold, determining that the moving target exists in the background; When the number of the edge pixels is less than the third number threshold, it is determined that the moving object does not exist in the background.

7. The method according to claim 1, characterized in that The obtaining of a first pose of the target vehicle at a first time and a second pose of the target vehicle at a second time includes: Calculating dead reckoning data of the target vehicle using a dead reckoning algorithm; The first position of the target vehicle at a first time and the second position of the target vehicle at a second time are determined based on the dead reckoning data.

8. A moving target detection device, characterized in that: The device comprises: an acquisition unit, configured to acquire a first pose of a target vehicle at a first time and a second pose of the target vehicle at a second time, wherein the first pose and the second pose are both poses in a world coordinate system, the first time is a timestamp of a previous frame, the second time is a timestamp of a current frame, and both the previous frame and the current frame include a background of the target vehicle; a first processing unit, configured to obtain a third pose of the target vehicle in a target pixel coordinate system based on the first pose and a transformation relationship, wherein the transformation relationship is a transformation relationship between the target pixel coordinate system and the world coordinate system, and the pixel coordinate system of the previous frame and the pixel coordinate system of the current frame are both the target pixel coordinate system; A second processing unit is configured to obtain a fourth posture of the target vehicle in the target pixel coordinate system according to the second posture and the conversion relationship; a third processing unit, configured to obtain a reference motion vector of the target vehicle from the first time to the second time according to the third posture and the fourth posture; The third processing unit is further configured to obtain a target motion vector of the background according to the reference motion vector and a relative motion relationship between the target vehicle and the background.

9. An electronic device, characterized in that: include: A processor and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 7.

Images