Target detection method and related device

Abstract

The invention discloses a target detection method and a related device, which can be applied to automatic driving or auxiliary driving. The method comprises the following steps: acquiring a plurality of images, wherein the images comprise a detection target; determining at least one key point on the contour of the detection target; determining a motion track of each key point; performing classification processing on the motion track of each key point through a plurality of classifiers to obtain a motion detection result of each key point; wherein the plurality of classifiers comprise a static motion classifier, a non-parallel motion classifier, a high-speed parallel motion classifier and a low-speed motion classifier; the plurality of classifiers further comprise one or more of the following items: a motion angle classifier, a motion scale classifier and a retrograde motion classifier; and determining a motion detection result of the detection target based on the motion detection result of each key point. Through the scheme, a more accurate motion detection result of the detection target can be obtained.

Description

A target detection method and related device Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a target detection method and related devices. Background Art

[0002] With the continuous development of intelligent driving, more and more vehicles are equipped with cameras, which can observe the environment around the vehicle and assist the driving system in making various driving decisions. For example, the camera monitors other moving objects in the vehicle's environment (such as vehicles, obstacles, pedestrians, etc.), and can estimate the motion state of these objects through calculation, thereby assisting the driving system in making various driving decisions (such as anti-collision braking, deceleration, etc.).

[0003] In order to estimate the motion state of other objects, a method for motion state judgment based on multi-frame optical flow is proposed at this stage. The principle of this method is: extract corner points on the target object, and the corner points on the images of different frames form multi-frame optical flow. The lowest and closest optical flow point to the ground is selected to calculate the motion distance of the target object between the two frames. If the motion distance is less than the motion distance of the vehicle itself, the target object is judged to be stationary, otherwise it is judged to be moving; then, the relative motion angle of the target object is calculated based on the optical flow convergence point and vanishing point of the optical flow on the target object, and then the motion state of the target object is determined.

[0004] However, in this method, corner points are extreme points in the image—points with particularly prominent attributes, such as isolated points with maximum or minimum intensity for certain attributes or the end points of line segments. The determination of corner points depends on the image quality and the corner point extraction algorithm. Therefore, the determined corner points may not effectively represent the overall motion state of the target object, resulting in large errors in the determined motion state. Furthermore, because this method approximates the lowest, closest corner points to the ground as corner points on the ground, the calculated motion distance of the target object may differ significantly from the actual motion distance of the target object.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a target detection method and related devices, which can obtain more accurate motion detection results of the detected target.

[0007] In a first aspect, the present application provides a method for detecting a target, the method comprising: acquiring multiple images, the image including the detection target; determining at least one key point on the outline of the detection target; determining the motion trajectory of each key point; classifying the motion trajectory of each key point through multiple classifiers to obtain a motion detection result for each key point; wherein the multiple classifiers include: a stationary motion classifier, a non-parallel motion classifier, a high-speed parallel motion classifier, and a low-speed motion classifier; the multiple classifiers also include one or more of the following: a motion angle classifier, a motion scale classifier, and a retrograde motion classifier; and determining the motion detection result of the detection target based on the motion detection result of each key point.

[0008] In this solution, the key points of the detection target are points with specific features on the contour of the detection target (for example, the key points of a vehicle can include the vehicle's rearview mirror, tire contact points, headlights, etc.). The motion trajectory calculated based on the key points is more consistent with the actual motion of the detection target than the motion trajectory calculated by optical flow, etc., and can provide more reliable input for subsequent motion detection. The judgment result of a classifier on a motion feature of a key point is relatively accurate. By combining the judgment results of multiple classifiers, a more accurate motion detection result of the key point can be obtained. Furthermore, by jointly determining the motion detection result of the detection target through the motion detection result of each key point, a more accurate motion detection result of the detection target can be obtained.

[0009] For example, the image may be a picture, a photo, a video, a streaming media, etc. The detection target may be a vehicle, a pedestrian, a building, a roadblock, etc. The detection target may be understood as a person or object that may affect the movement of the mobile station.

[0010] In conjunction with the first aspect, in one possible implementation, the motion detection result includes a motion state and a first motion probability, and the first motion probability is used to indicate the probability of being in the motion state. Exemplarily, the first motion probability can be a confidence level. Through the motion detection result, the motion state of the detection target and the probability that the detection target is in the motion state can be determined. Furthermore, the motion detection result can be used as input data for other devices or apparatuses. Exemplarily, it can be used as input for an adjustment module of a mobile station (e.g., an advanced driver assistance system, a vehicle controller, etc.), providing it with accurate motion detection results of the detection target, which can better assist the driving of the mobile station.

[0011] In conjunction with the first aspect, in one possible implementation, one classifier is used to determine a motion feature of the key point and a second motion probability of the key point being within the motion feature; the motion state is determined based on multiple motion features, and the first motion probability is determined based on multiple second motion probabilities. A single classifier can provide a relatively accurate result for determining a single motion feature of a key point. Combining the results of multiple classifiers can yield a more accurate motion detection result for the key point.

[0012] In conjunction with the first aspect, in one possible implementation, the motion state includes one or more of the following: forward fast motion state, forward slow motion state, stationary state, forward cut-in state, forward cut-out state, high-speed reverse motion state, low-speed reverse motion state, forward accelerated motion state, forward decelerated motion state, forward rapid accelerated motion state, forward rapid decelerated motion state, emergency stop state, crossing state, parallel motion state, lateral turning state, longitudinal motion state, U-turn state, and parameter information describing the motion state; wherein the parameter information describing the motion state includes one or more of the following: absolute speed information, relative speed information, and collision time information. In this embodiment, the determined motion state has a more detailed feature description than simple motion states such as stationary and moving, or simple speed and acceleration information. In this way, more effective and accurate motion detection results can be provided.

[0013] In conjunction with the first aspect, in one possible implementation, the motion features determined by the stationary motion classifier include: stationary and moving; the motion features determined by the non-parallel motion classifier include: lateral driving, cutting in, and cutting out; the motion features determined by the high-speed parallel motion classifier include: fast, high speed, acceleration, and rapid acceleration; the motion features determined by the low-speed motion classifier include: slow, deceleration, and rapid deceleration; the motion features determined by the motion angle classifier include: judging turning, U-turn, parallel driving, and forward / reverse motion; the motion features determined by the motion scale classifier include: forward / reverse acceleration; and the motion features determined by the reverse motion classifier include: reverse driving and forward driving. Multiple motion classifiers utilize information such as pixel scale changes, true scale changes, and angle changes of the detected target, and are more robust than methods that use multi-frame optical flow to determine motion states.

[0014] In conjunction with the first aspect, in one possible implementation, the motion detection result of the detection target corresponds to a first moment, and the method further includes: updating the motion detection result of the detection target corresponding to the first moment based on the motion detection result of the detection target corresponding to a second moment and the motion detection result of the detection target, where the second moment is before the first moment. In this way, a more accurate motion detection result can be determined based on the continuity of the detection target's motion.

[0015] In conjunction with the first aspect, in one possible implementation, the camera that captures the multiple images is located on a mobile stage, and the motion trajectory of each key point is classified using multiple classifiers to obtain a motion detection result for each key point. This includes: obtaining motion parameters of the mobile stage; and, based on the motion parameters of the mobile stage, classifying the motion trajectory of each key point using multiple classifiers to obtain a motion detection result for each key point. In this way, by taking into account the motion parameters of the mobile stage, the influence of the mobile stage's own motion on the determination of the motion state of the detection target can be eliminated, and the detection module can obtain more accurate motion detection results.

[0016] In combination with the first aspect, in one possible implementation, determining at least one key point on the contour of the detection target includes: performing target detection on the area of ​​interest in the image to determine a bounding box containing the detection target; processing the image in the bounding box to determine the contour of the detection target and the type of the detection target, which type includes one or more of the following: vehicle, pedestrian, building, roadblock; determining at least one key point on the contour of the detection target based on the contour and the type, and the position of the key point on the contour is determined based on the type.

[0017] Among them, the key points of the detection target are points with specific characteristics on the outline of the detection target. By determining the type of the detection target, the position of the key points on the outline of the detection target can be determined more conveniently, which can improve the accuracy of determining the key points and reduce the processing time. By setting the region of interest, the area that needs to be focused on can be determined from the original image, reducing processing time and increasing accuracy. Furthermore, the bounding box of the detection target can contain the detection target, which is an image area that is smaller than the region of interest. By setting the bounding box, the range of the image that needs to be processed can be further reduced, the processing time can be reduced, and the accuracy can be increased.

[0018] In conjunction with the first aspect, in one possible implementation, the camera that acquires the multiple images is located on a mobile platform, and the method further includes: determining at least one feature coordinate point within the bounding box; classifying the motion trajectory of each feature coordinate point using a bounding box motion classifier to obtain a relative speed of the detection target relative to the mobile platform and a third motion probability of being at that relative speed; and updating the motion detection result of the detection target based on the relative speed and the third motion probability. In this way, since the feature coordinate points on the bounding box are easy to find and the bounding box has a strong correlation with the detection target, the motion characteristics of the bounding box can better reflect the motion characteristics of the detection target. Therefore, when there are fewer motion trajectories of key points, the bounding box motion classifier can obtain more accurate judgment results.

[0019] In combination with the first aspect, in one possible implementation, the multiple images include a first image at a third moment and a second image at a fourth moment, and determining the motion trajectory of each key point includes: determining the first position information of each key point in the first image; determining the second position information of each key point in the second image; and determining the motion trajectory of each key point based on the first position information and the second position information. Optionally, the motion trajectory can also be referred to as a motion field, a motion vector, etc. In order to obtain more information, the motion trajectory of the key point can also be determined by using images taken at more moments, which is not limited in the embodiments of the present application.

[0020] In conjunction with the first aspect, in a possible implementation, the multiple classifiers are connected in a cascade or average integration manner. Optionally, the multiple classifiers can also be connected in a partial cascade or partial average integration manner. In this cascade connection mode, the determination results obtained by each classifier can serve as a reference for each other. In this way, the correlation between the determination results of different classifiers for a certain class or several classes of motion states can be fully utilized. In this average integration connection mode, the determination results obtained by each classifier are obtained separately. If the determination result of a classifier is not very accurate, it will not affect the determination results of other classifiers. In this partial cascade or partial average integration connection mode, classifiers with strong correlation can be connected in a cascade manner, and classifiers with weaker correlation can be connected in an average integration manner.

[0021] In the second aspect, the present application provides a detection device, which includes an acquisition unit, a first determination unit, a second determination unit, a processing unit and a third determination unit: the acquisition unit is used to acquire multiple images, which include a detection target; the first determination unit is used to determine at least one key point on the contour of the detection target; the second determination unit is used to determine the motion trajectory of each key point; the processing unit is used to classify the motion trajectory of each key point through multiple classifiers to obtain a motion detection result for each key point; wherein the multiple classifiers include: a stationary motion classifier, a non-parallel motion classifier, a high-speed parallel motion classifier, and a low-speed motion classifier; the multiple classifiers also include one or more of the following: a motion angle classifier, a motion scale classifier, and a retrograde motion classifier; the third determination unit is used to determine the motion detection result of the detection target based on the motion detection result of each key point.

[0022] In combination with the second aspect, in a possible implementation, the motion detection result includes a motion state and a first motion probability, where the first motion probability is used to indicate a probability of being in the motion state.

[0023] In combination with the second aspect, in one possible implementation, a classifier is used to determine a motion feature of the key point and a second motion probability of the key point being in the motion feature; the motion state is determined based on multiple motion features, and the first motion probability is determined based on multiple second motion probabilities.

[0024] In combination with the second aspect, in a possible implementation method, the motion state includes one or more of the following: forward fast motion state, forward slow motion state, stationary state, forward cut-in state, forward cut-out state, high-speed reverse motion state, low-speed reverse motion state, forward acceleration state, forward deceleration state, forward rapid acceleration state, forward rapid deceleration state, emergency stop state, crossing state, parallel motion state, lateral turning state, longitudinal motion state, U-turn state, and parameter information describing the motion state; wherein, the parameter information describing the motion state includes one or more of the following: absolute speed information, relative speed information, and collision time information.

[0025] In combination with the second aspect, in a possible implementation method, the motion features determined by the stationary motion classifier include: stationary and moving; the motion features determined by the non-parallel motion classifier include: lateral driving, cutting in, and cutting out; the motion features determined by the high-speed parallel motion classifier include: fast, high speed, acceleration, and rapid acceleration; the motion features determined by the low-speed motion classifier include: slow, deceleration, and rapid deceleration; the motion features determined by the motion angle classifier include: judging turning, U-turn, parallel driving, and forward / reverse motion; the motion features determined by the motion scale classifier include: forward / reverse acceleration; the motion features determined by the reverse motion classifier include: reverse driving and forward driving.

[0026] In combination with the second aspect, in one possible implementation method, the motion detection result of the detection target corresponds to the first moment, and the third determination unit is also used to: update the motion detection result of the detection target corresponding to the first moment based on the motion detection result of the detection target corresponding to the second moment and the motion detection result of the detection target, and the second moment is before the first moment.

[0027] In combination with the second aspect, in one possible implementation, the camera that acquires the multiple images is located on a mobile platform, and the acquisition unit is also used to: acquire the motion parameters of the mobile platform; the processing unit is specifically used to: based on the motion parameters of the mobile platform, classify the motion trajectory of each key point through multiple classifiers to obtain the motion detection result of each key point.

[0028] In combination with the second aspect, in one possible implementation, the first determination unit is specifically used to: perform target detection on the area of ​​interest in the image, and determine a bounding box containing the detected target; process the image in the bounding box to determine the outline of the detected target and the type of the detected target, which type includes one or more of the following: vehicle, pedestrian, building, roadblock; determine at least one key point on the outline of the detected target based on the outline and the type, and the position of the key point on the outline is determined based on the type.

[0029] In combination with the second aspect, in one possible implementation, the camera that acquires the multiple images is located on a mobile platform, and the third determination unit is further used to: determine at least one feature coordinate point in the bounding box; classify the motion trajectory of each feature coordinate point through a bounding box motion classifier to obtain the relative speed of the detection target relative to the mobile platform, and the third motion probability at the relative speed; and update the motion detection result of the detection target based on the relative speed and the third motion probability.

[0030] In combination with the second aspect, in a possible implementation, the multiple images include a first image at a third moment and a second image at a fourth moment, and the second determination unit is specifically used to: determine the first position information of each key point in the first image; determine the second position information of each key point in the second image; and determine the motion trajectory of each key point based on the first position information and the second position information.

[0031] In combination with the second aspect, in a possible implementation, the multiple classifiers are connected by cascading or averaging integration.

[0032] In a third aspect, the present application provides a processing device comprising a processor and a memory; the memory is used to store program code; and the processor is used to call the program code from the memory to execute the method described in the first aspect or any possible implementation of the first aspect.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium for storing instructions. When the instructions are executed, the method described in the first aspect or any possible implementation of the first aspect is implemented.

[0034] The beneficial effects of each possible implementation method in the second to fourth aspects can be found in the corresponding description in the first aspect and will not be repeated here.

[0035] Through the solution of the present application, the motion trajectory of at least one key point on the detection target is processed by multiple classifiers to obtain the motion detection result of each key point, and then the motion detection result of the detection target is determined based on the motion detection result of each key point. Among them, the motion trajectory calculated based on the key point is less time-consuming and more accurate than the motion trajectory calculated by optical flow, etc., and is more consistent with the actual movement of the detection target, providing a more reliable input for subsequent motion detection; in addition, multiple motion classifiers utilize information such as pixel scale changes, real scale changes, and angle changes of the detection target. Compared with the method of using multi-frame optical flow to judge the motion state, it has higher robustness and the obtained motion detection result is more accurate. In summary, through the embodiment of the present application, more accurate motion detection results of the detection target can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0037] FIG1 is a schematic structural diagram of a mobile station system provided in an embodiment of the present application;

[0038] FIG2 is a schematic flow chart of a target detection method provided in an embodiment of the present application;

[0039] FIG3 is a schematic diagram of a key point detection target provided by an embodiment of the present application;

[0040] FIG4 is a schematic diagram of a region of interest of an image and a bounding box of a detection target provided by an embodiment of the present application;

[0041] FIG5 is a schematic diagram of the connection relationship of some multiple classifiers provided in an embodiment of the present application;

[0042] FIG6 is a schematic diagram of a detection device provided in an embodiment of the present application;

[0043] FIG7 is a schematic diagram of a detection device provided in an embodiment of the present application;

[0044] FIG8 is a schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application are described in more detail below.

[0046] Referring to Figure 1, there is shown a schematic diagram of the structure of a mobile station system provided in an embodiment of the present application. The mobile station can be a car, a robot, a drone, etc. The system includes an input module, a sensor, a detection module, and an adjustment module. It should be noted that the modules shown in Figure 1 are merely examples. In actual application scenarios, the mobile station may include more or fewer modules or components, and this embodiment of the present application does not impose any limitations. The following further describes each module or component.

[0047] The input module is used to receive control information for the mobile station. For example, the control information includes steering wheel angle information, driving gear information, driving and braking information, etc.

[0048] Sensors: Based on the object they measure, sensors can be divided into two categories: one that measures the state of the mobile station itself, and the other that measures the state of the mobile station's environment. The former can include the Global Positioning System (GPS), velocity sensors, accelerometers, angular velocity sensors, torque sensors, and so on. Furthermore, the mobile station may also contain sensors that collect more information related to the mobile station's motion (or driving or flight), such as engine operating conditions, operating temperatures of various modules or components, intake pressure, intake temperature, and so on. The latter can include cameras, lidar sensors, millimeter-wave radar sensors, ultrasonic sensors, and so on. These sensors can collect information related to the mobile station's environment. The various types of information collected by the sensors can be converted into electrical signals and transmitted to the mobile station's adjustment module. After analysis, the adjustment module outputs control / adjustment instructions for the mobile station or provides operational suggestions to the user to assist in controlling the mobile station.

[0049] Specifically, a camera (or video camera, or camera) serves as a sensor that can capture images of the mobile station's surroundings to collect information about the mobile station's environment. Exemplarily, the camera is used to collect road condition information in front of the mobile station while the mobile station is in motion. In other exemplary embodiments, the camera is also used to capture reverse images of the mobile station, driving recordings, and the like. Optionally, the camera can be a monocular camera, a binocular camera, a multi-camera camera, a wide-angle camera, or a surround-view camera.

[0050] The detection module is configured to determine the motion state of other objects in the environment based on information about the mobile station's environment captured by the camera. The detection results determined by the detection module can be transmitted to the adjustment module, which can then control / adjust the mobile station's driving state based on the detection results or provide operational suggestions to the user.

[0051] An adjustment module is used to control / adjust the driving state of the mobile station. Exemplarily, the adjustment module can be an advanced driver assistance system (ADAS), a vehicle control unit (VCU), an electronic stability program (ESP), or the like. For example, an ADAS can use environmental data collected by sensors such as cameras, radar, lasers, and ultrasonic waves inside and outside the vehicle to identify static and dynamic objects, and utilize detection and tracking techniques to perceive the vehicle's driving intent. The VCU is the control unit that implements vehicle control decisions. It determines the driver's driving intent by collecting signals such as the accelerator pedal, gear position, brake pedal, and steering wheel angle. By monitoring vehicle status information (such as speed and temperature), the VCU determines and processes the information and sends vehicle operating status control commands to the power system and power battery system, while also controlling the operating mode of the vehicle's accessory power system. The VCU also provides fault diagnosis, protection, and storage functions for the entire vehicle system. ESP can help the vehicle maintain dynamic balance by analyzing the vehicle's driving status information sent from vehicle sensors and then issuing correction instructions to the anti-lock brake system (ABS), electronic brake distribution (EBD), etc.

[0052] In the embodiment of the present application, the adjustment module can obtain information collected by various sensors and the detection results obtained by the detection module, analyze them, and output control instructions for the mobile station or provide some operational suggestions to the user to assist the user in controlling the mobile station. Therefore, the accuracy of the detection results determined by the detection module will affect the control decisions output by the adjustment module.

[0053] Optionally, the detection module may be an independent module or a component integrated into the adjustment module, which is not limited in the embodiment of the present application.

[0054] In other possible implementations, the embodiments of the present application can also be applied to distributed sensor networks or non-mobile platforms, such as streetlights and traffic lights. Related fields include smart intersections and smart cities. For example, cameras installed on streetlights and traffic lights can be used to detect the motion status of obstacles in traffic areas.

[0055] The target detection method and detection device provided in this application can improve the accuracy of determining the motion state of the detected target. When this method or device is applied to a terminal, it can improve the terminal's perception capability.

[0056] Referring to Figure 2, it is a flow chart of a method for detecting a target provided in an embodiment of the present application. The method can be implemented based on the system shown in Figure 1. Optionally, the execution subject of the method described below can be the detection module in Figure 1, which can be an independent module, a component integrated in the adjustment module, or a component integrated in the sensor. Optionally, the execution subject of the method described below can also be other modules in the terminal to implement the function of determining the motion detection result of the detection target introduced in the following method. The method includes but is not limited to the following steps.

[0057] S101. Acquire multiple images, where the images include detection targets.

[0058] In one possible implementation, the multiple images can be acquired by a camera configured on a terminal (e.g., a mobile station). For example, the camera installed on the mobile station can capture images of the environment in which the mobile station is located (the multiple images can be multiple frames of images captured continuously) and send these images to the detection module. The detection module processes the multiple images received to identify images containing the detection target. For example, the images can be pictures, photos, videos, streaming media, etc.

[0059] Exemplarily, the detection target may be a vehicle, pedestrian, building, roadblock, etc. A roadblock refers to an obstacle placed on the road to warn pedestrians or vehicles or to convey road condition information. In the embodiments of the present application, the detection target may be understood as a person or object that may affect the movement of the mobile station.

[0060] S102: Determine at least one key point on the outline of the detection target.

[0061] Optionally, at least one key point on the contour of the detection target can be determined by a detection network model (or algorithm system) that can identify the key points of the detection target. Next, a possible method for generating a detection network model is introduced. First, the designer obtains training data. For example, a number of pictures containing various detection targets (for example, vehicles, pedestrians, buildings, roadblocks, etc.) can be taken in a typical traffic scene by a camera of a mobile station, and the key points (and also the bounding boxes) of various detection targets appearing in the captured images can be marked to obtain a training data set. The training data set includes the original image and the key points on the detection target. Afterwards, the network structure of the detection network model is designed and the model training is performed. The detection network model can identify images containing detection targets. Optionally, the detection network model can also output the bounding box of the detection target and at least one key point on the contour of the detection target.

[0062] Exemplarily, referring to FIG3 , is a schematic diagram of the key points of a detection target provided by an embodiment of the present application. The detection target illustrated in FIG3 is a vehicle, and the figure illustrates 36 possible key points on the vehicle (only as an example, there may be more or fewer in actual applications). During the model training process, these key points can be annotated. The key points include the vehicle's rearview mirrors, tire contact points, headlights, and other key feature points of the vehicle. It will be understood that for different types of detection targets, the position settings of the key points are different. For example, if the detection target is a person, the key points can be the top of the head, shoulders, foot contact points, end of the hands, and other key feature points of the person.

[0063] In one possible implementation, at least one key point on the contour of the detection target is determined by: performing target detection on a region of interest (ROI) in an image to determine a bounding box (or a bounding box) containing the detection target; processing an image in the bounding box to determine the contour of the detection target and the type of the detection target, where the type includes one or more of the following: vehicle, pedestrian, building, and roadblock; and determining at least one key point on the contour of the detection target based on the contour and the type, where the position of the key point on the contour is determined based on the type.

[0064] Among them, the region of interest is an image area selected from the original image. For example, the region of interest can be a rectangular area of ​​a preset size located in the center of the original image. In addition, the region of interest can also be determined by other operators and functions, and the embodiments of the present application are not limited. By setting the region of interest, the area that needs to be focused on can be determined from the original image, reducing processing time and increasing accuracy. Furthermore, the bounding box of the detection target can contain the detection target and is an image area that is smaller than the region of interest. Optionally, one detection target can correspond to one bounding box. Optionally, the bounding box can be a two-dimensional or three-dimensional bounding box. By setting the bounding box, the image range that needs to be processed can be further reduced, processing time can be reduced, and accuracy can be increased. For example, see Figure 4, which is a schematic diagram of the region of interest of an image and the bounding box of the detection target provided in an embodiment of the present application.

[0065] Optionally, in actual application, the types of detection targets may be more or fewer, or a type may be further divided. For example, for vehicles, they can be further divided into trucks, cars, motorcycles, bicycles, and so on. For different types, the position of the key point of the detection target may be different, that is, the position of the key point on the contour of the detection target is determined based on the type of the detection target. Optionally, the correspondence between the position of the key point on the contour of the detection target and the type of the detection target can be targeted during the training process of the detection network model, so that the detection network model can better handle various types of detection targets and improve the accuracy of judgment.

[0066] S103: Determine the motion trajectory of each key point.

[0067] In some embodiments, the image is a multi-frame image captured continuously, and the motion trajectory of the key point is determined based on the position information of the same key point in the images of different frames. Optionally, the motion trajectory can also be called a motion field, a motion vector, etc.

[0068] In one possible implementation, the multiple images include a first image at a third moment, and a second image at a fourth moment, and determining the motion trajectory of each key point includes: determining the first position information of each key point in the first image; determining the second position information of each key point in the second image; and determining the motion trajectory of each key point based on the first position information and the second position information. The position information can be absolute coordinate information or relative coordinate information. The motion trajectory of the key point can reflect the various motion characteristics of the key point. For example, if the first position information and the second position information are the same, it can be indicated that the key point has not moved and is in a stationary state. If the first position information and the second position information are different, it can be indicated that the key point has moved, and the motion trajectory of the key point can be determined, and further the motion speed, angle, and other information can be calculated.

[0069] It should be noted that, in the embodiment of the present application, the motion trajectory of the key point is determined by at least two images taken at different times. In order to obtain more information, the motion trajectory of the key point can also be determined by images taken at more times, which is not limited by the embodiment of the present application. In addition, in a possible implementation method, the characteristic coordinate point on the bounding box can also be used as a special key point, and the motion trajectory of the characteristic coordinate point can also be used as the basis for determining the motion detection result of the detection target. The characteristic coordinate point can be the vertex of the bounding box, the midpoint of the bounding box, or the midpoint of the border of the bounding box, etc.

[0070] The motion trajectory calculated based on key points in the embodiment of the present application is less time-consuming, more accurate, and more consistent with the actual movement of the detection target than the motion trajectory calculated by optical flow. In this way, more reliable input can be provided for subsequent motion detection.

[0071] S104 , classifying the motion trajectory of each key point through multiple classifiers to obtain a motion detection result for each key point.

[0072] Among them, the multiple classifiers include: a stationary motion classifier, a non-parallel motion classifier, a high-speed parallel motion classifier, and a low-speed motion classifier; the multiple classifiers also include one or more of the following: a motion angle classifier, a motion scale classifier, and a retrograde motion classifier.

[0073] Specifically, a classifier is used to determine a motion feature of a key point, and the probability of the key point being in a second motion of the motion feature. Each classifier is further introduced below. It should be noted that, in addition to the 7 types of motion classifiers introduced below, the multiple classifiers can also include other classifiers with motion feature classification functions, and the embodiment of the present application does not limit the number of classifiers included. In addition, two or more partial classifiers among these multiple classifiers can also be combined to form a classifier with more comprehensive functions, or a classifier can also be split into multiple classifiers with more detailed and specific judgment rules. The splitting and combination of classifiers are not used to limit the scope of the embodiment of the present application.

[0074] Optionally, the functions of the multiple classifiers can be implemented based on a neural network. Exemplarily, the neural network can be a convolutional neural network (CNN), a long short-term memory network (LSTM), or a convolutional long short-term memory network (conv-LSTM), etc. Exemplarily, the neural network can be composed of multiple processing layers or multiple long short-term memory network (LSTM) units. Among them, the functions implemented by some processing layers or some LSTM units that constitute the neural network can correspond to the functions implemented by one of the multiple classifiers. When the amount of data to be processed is large, the processing results of the neural network will be more accurate than the processing results of multiple classifiers.

[0075] 1. The motion features determined by the stationary motion classifier include: stationary and moving. The stationary motion classifier can also derive the probability that the key point is in the second motion of the motion feature, which can be expressed, for example, by confidence. Among them, confidence can also be called reliability, confidence level, and confidence coefficient, and confidence indicates how credible a certain estimation result is. Exemplarily, the stationary motion classifier classifies the motion trajectory of a key point and derives that the key point is stationary, and the confidence level of the key point is 85%, which means that the key point has an 85% probability (or probability) of being stationary. The stationary motion classifier is more accurate in determining whether the key point is in a stationary / moving state.

[0076] 2. The motion features determined by the non-parallel motion classifier include: lateral movement, cut-in movement, and cut-out movement. The non-parallel motion classifier can also derive the second motion probability of the key point being in the motion feature, which can be represented by a confidence level, for example. Among them, lateral movement indicates that the motion direction of the detection target is approximately perpendicular to the motion direction of the mobile station itself, cut-in movement indicates that the key point enters the motion path of the mobile station, and cut-out movement indicates that the key point leaves the motion path of the mobile station. The motion angle classifier is more accurate in determining the motion direction of the key point.

[0077] 3. The motion features determined by the high-speed parallel motion classifier include: rapid motion, high-speed motion, accelerated motion, and rapidly accelerated motion. The high-speed parallel motion classifier can also derive the probability of the key point being in the second motion of the motion feature, which can be represented by a confidence level. Optionally, the determination result of the high-speed parallel motion classifier can be determined based on the calculated speed and acceleration of the key point. When the key point is in high-speed motion, the determination result of the high-speed parallel motion classifier is more accurate.

[0078] 4. The motion features determined by the low-speed motion classifier include: slow motion, decelerated motion, and rapidly decelerated motion. The low-speed motion classifier can also derive the probability of the key point being in the second motion of the motion feature, which can be represented by a confidence level, for example. Optionally, the determination result of the low-speed parallel motion classifier can be determined based on the calculated velocity and acceleration of the key point. When the key point is in slow motion, the determination result of the low-speed motion classifier is relatively accurate.

[0079] 5. The motion features determined by the motion angle classifier include: turning, U-turn, parallel driving, and forward / reverse driving. The motion angle classifier can also derive the second motion probability of the key point being in the motion feature, which can be expressed, for example, by confidence. The motion angle classifier is more accurate in determining the angular velocity of the key point. In particular, if the detection target is close to the terminal, the determination result obtained by the motion angle classifier is more accurate. It should be noted that the motion corner classifier can determine the motion angle of the key point based on the relationship between the convergence point generated by the key point and the optical flow convergence point of the static scene, and then calculate the angular velocity of the key point. In this way, since the key point of the detection target is a point with specific features on the outline of the detection target, it has a stronger correlation with the detection target. As the basis for determining the motion angle, the convergence point generated based on the key point will be more accurate than the optical flow convergence point of the optical flow on the detection target. In this way, the accuracy of determining the motion angle of the key point can be improved.

[0080] 6. The motion features determined by the motion scale classifier include forward and reverse acceleration. The motion scale classifier can also determine the probability that the key point is in the second motion of the motion feature, which can be expressed, for example, as a confidence level. It should be noted that the higher the confidence level, the greater the acceleration of the key point. The motion scale classifier's determination of the acceleration of the key point is relatively accurate. In particular, if the detected target is far away from the terminal, the determination result obtained by the motion scale classifier is relatively accurate.

[0081] 7. The motion features determined by the retrograde motion classifier include: reverse travel and forward travel. The retrograde motion classifier can also determine the probability of the key point being in the second motion of the motion feature, which can be represented by a confidence level, for example. The retrograde motion classifier provides relatively accurate results in determining whether the key point is in reverse travel or forward travel.

[0082] The multiple motion classifiers introduced above utilize information about the pixel scale change, true scale change, angle change, etc. of the detected target. Compared with the method of using multi-frame optical flow to judge the motion state, its robustness is higher.

[0083] The multiple classifiers can be used to obtain various motion features of the key point. These multiple motion features can collectively determine the motion state of the key point, and the multiple second motion probabilities can collectively determine a first motion probability that the key point is in a certain motion state. In other words, the motion state of the key point is determined based on the multiple motion features, and the first motion probability is determined based on the multiple second motion probabilities of the key point. The motion detection result of the key point includes the motion state of the key point and the first motion probability.

[0084] The motion state is further introduced below. In an embodiment of the present application, the motion state includes one or more of the following: forward fast motion state, forward slow motion state, stationary state, forward cut-in state, forward cut-out state, high-speed reverse motion state, low-speed reverse motion state, forward acceleration state, forward deceleration state, forward rapid acceleration state, forward rapid deceleration state, emergency stop state, crossing state, parallel motion state, lateral turning state, longitudinal motion state, U-turn state, and parameter information describing the motion state. Among them, the parameter information describing the motion state includes one or more of the following: absolute speed information, relative speed information, and collision time (TTC) information. The collision time is used to indicate the time required for the current mobile station to collide with the detection target when the motion states of both parties remain unchanged.

[0085] For example, the motion state determined by motion characteristics such as motion, rapid motion, and forward driving may be a "forward rapid motion state." The motion state determined by motion characteristics such as motion, slow motion, and forward driving may be a "forward slow motion state." The motion state determined by motion characteristics such as motion, rapid deceleration motion, and forward driving may be a "forward rapid deceleration motion state." The motion state determined by motion characteristics such as motion and motion angle may be a "U-turn state." There may be other possible ways to determine the motion state based on multiple motion characteristics, which are not listed one by one in the embodiments of the present application.

[0086] For the first motion probability of the key point, it can represent the probability that a key point is in a certain determined motion state, and can provide a basis for the subsequent determination of the first motion probability of the detection target. Optionally, the first motion probability can be the arithmetic mean or weighted average of the multiple second motion probabilities. It should be noted that the first motion probability may be calculated from part of the second motion probabilities. For example, the second motion probability that is too small can be filtered out. For example, if the second motion probability of slow motion is 5%, it means that the possibility that the key point is in slow motion is very small. When calculating the first motion probability of the detection target, the second motion probability of slow motion can be ignored.

[0087] Optionally, the multiple classifiers can be connected in a cascade or average integration manner. Optionally, the multiple classifiers can also be connected in a partial cascade or partial average integration manner. Exemplarily, referring to Figure 5, it is a schematic diagram of the connection relationship of some multiple classifiers provided in an embodiment of the present application. The classifiers included in Figure 5 are: a stationary motion classifier, a non-parallel motion classifier, a high-speed parallel motion classifier, a low-speed motion classifier, a motion angle classifier, a motion scale classifier, and a retrograde motion classifier. It should be noted that in actual applications, the multiple classifiers may include one or more of a motion angle classifier, a motion scale classifier, and a retrograde motion classifier.

[0088] Among them, (1) in Figure 5 illustrates a schematic diagram of connecting multiple classifiers in a cascade manner. In this cascade connection mode, the judgment results obtained by each classifier can serve as a reference for each other. In this way, the correlation between the judgment results of different classifiers for a certain type or several types of motion states can be fully utilized. For example, for forward accelerated motion, the judgment results obtained by the high-speed parallel motion classifier (which can determine accelerated motion) and the motion scale classifier (which can determine acceleration) have a strong correlation and can serve as a reference for each other to obtain more accurate results. (2) in Figure 5 illustrates a schematic diagram of connecting multiple classifiers in an average integration manner. In this average integration connection mode, the judgment results obtained by each classifier are obtained separately. If the judgment result of one classifier is not very accurate, it will not affect the judgment results of other classifiers. (3) in Figure 5 illustrates a schematic diagram of connecting multiple classifiers in a partial cascade and partial average integration manner. Optionally, classifiers with strong correlation can be connected in a cascade manner, and classifiers with weaker correlation can be connected in an average integration manner. It should also be noted that the connection order of each classifier can be adjusted according to the needs of actual application, and this application does not impose any restrictions.

[0089] In one possible implementation, the motion detection result for each key point is further determined based on the motion parameters of the mobile station. That is, the method further includes: obtaining the motion parameters of the mobile station. Based on the motion parameters of the mobile station, the motion trajectory of each key point is classified using multiple classifiers to obtain a motion detection result for each key point. The motion parameters of the mobile station may include information such as the speed, acceleration, and angular velocity of the mobile station, and may be obtained using various sensors of the mobile station, such as GPS, velocity sensors, acceleration sensors, angular velocity sensors, torque sensors, inertial measurement units (IMUs), global navigation satellite systems (GNSSs), and speedometers (WSSs). It should be noted that in this implementation, the motion parameters of the mobile station and the motion trajectory of the key points serve as inputs to the multiple motion classifiers, jointly determining the output results of the motion classifiers. In this way, by taking into account the motion parameters of the mobile station, the influence of the mobile station's own motion on the determination of the motion state of the detection target can be eliminated, thereby obtaining a more accurate motion detection result.

[0090] In one possible implementation, when determining the motion detection results of key points, the weights of the multiple classifiers may differ. Alternatively, the weights of the multiple classifiers may be obtained through training using a boosting algorithm. Alternatively, the weights of the multiple classifiers may take on different values ​​in different driving scenarios to adapt to different driving scenarios and obtain more accurate determination results. Exemplarily, the driving scenarios may include straight driving, curve driving, uphill driving, downhill driving, and the like.

[0091] S105 : Determine the motion detection result of the detection target based on the motion detection result of each key point.

[0092] The motion detection result of the detection target includes the motion state of the detection target and a first motion probability, where the first motion probability is used to indicate the probability of the detection target being in a certain motion state.

[0093] Optionally, the first motion probability and motion state of the detection target can be the first motion probability with the largest value in the motion detection result of each key point and its corresponding motion state. Exemplarily, there are three key points, wherein the motion state of the first key point is a forward rapid motion state, and the first motion probability is 80%; the motion state of the second key point is a forward rapid motion state, and the first motion probability is 85%; the motion state of the third key point is a forward accelerated motion state, and the first motion probability is 70%; then, the motion state of the detection target can be determined to be a forward rapid motion state (the motion state corresponding to the first motion probability with the largest value), and the first motion probability of the detection target is 85%.

[0094] Optionally, the motion detection result of the detection target may also be a motion detection result obtained by processing the motion detection result of each key point through a filter (or other algorithm system). Exemplarily, it may be a Kalman filter. Kalman filtering is an algorithm that uses a linear system state equation to optimally estimate the system state through system input and output observation data. Since the observation data includes the influence of noise and interference in the system, the optimal estimation can also be regarded as a filtering process. Data filtering is a data processing technology that removes noise and restores real data. Kalman filtering can estimate the state of a dynamic system from a series of data with measurement noise when the measurement variance is known, and can update and process the data collected on site in real time.

[0095] In some embodiments, the motion detection result of the detection target can also be combined with the motion features determined based on the bounding box of the detection target to jointly determine a more accurate motion detection result of the detection target. Optionally, the method also includes: determining at least one characteristic coordinate point in the bounding box; classifying the motion trajectory of each characteristic coordinate point using a bounding box motion classifier to obtain the relative speed of the detection target relative to the mobile station and a third motion probability of being at this relative speed; and updating the motion detection result of the detection target based on the relative speed and the third motion probability.

[0096] The bounding box is a bounding box of a preset shape that can contain the detected target. Exemplarily, the bounding box can be a rectangle. For an introduction to the bounding box, please refer to the description of FIG. 4 above. The characteristic coordinate points can be easily found in the preset graphic and have a unique positional relationship relative to the bounding box. Exemplarily, they can be vertices, center points, midpoints of the sides of the bounding box, and so on. The motion trajectory of the characteristic coordinate points can refer to the motion trajectory of the key points described above and will not be repeated here. Specifically, the bounding box motion classifier can determine the relative velocity of the detected target relative to the mobile station and a third motion probability of being at that relative velocity based on the motion trajectory of each characteristic coordinate point. Exemplarily, the third motion probability can be represented by a confidence level. It should be noted that a higher confidence level indicates a greater relative velocity of the detected target. The bounding box motion classifier's determination of the relative velocity of the detected target is more accurate. Optionally, the relative velocity of the detected target in the motion detection result of the detected target can be updated based on the relative velocity and the third motion probability.

[0097] Since the key points are points with specific features on the contour of the detection target, some key points may be obscured during the acquisition process, while the feature coordinate points on the bounding box are easy to find, and the bounding box has a strong correlation with the detection target. The motion features of the bounding box can better reflect the motion features of the detection target. Therefore, when there are fewer motion trajectories of key points, the bounding box motion classifier can obtain more accurate judgment results.

[0098] In some embodiments, the motion detection result of the detection target obtained at the current moment can also be combined with the motion detection result of the detection target obtained at the previous moment to jointly determine a more accurate motion detection result of the detection target. Exemplarily, the motion detection result of the detection target corresponds to the first moment. It can be understood that the multiple images used as the detection basis correspond to the first moment. For example, the multiple images are images taken within the minute of 14:01-14:02, and the first moment is 14:01-14:02. The method of the embodiment of the present application also includes: updating the motion detection result of the detection target corresponding to the first moment based on the motion detection result of the detection target and the motion detection result of the detection target corresponding to the second moment, and the second moment is before the first moment. Exemplarily, the second moment can be 14:00-14:01, and the motion detection result of the detection target corresponding to the second moment can be determined based on the motion detection result determined based on the images taken within the minute of 14:00-14:01 and the motion detection result of the detection target obtained at the previous moment of the second moment.

[0099] Optionally, the second moment may also be multiple moments before the first moment. For example, the second moment may be 13:58-14:01, and the motion detection results of the detection target corresponding to the second moment include the motion detection results of the detection target corresponding to the three moments 13:58-13:59, 13:59-14:00, and 14:00-14:01 respectively. Taking a specific example, if the motion detection result of the detection target corresponding to 13:58-13:59 is "86% in a forward accelerated motion state", the motion detection result of the detection target corresponding to 13:58-13:59 is "89% in a forward rapidly accelerated motion state", and the motion detection result of the detection target corresponding to 14:00-14:01 is "92% in a forward rapidly accelerated motion state", and the motion detection result of the detection target obtained at the current moment 14:01-14:02 is "83% in a forward rapidly accelerated motion state", the above motion detection results are input into the trained temporal neural network, and the updated (also understood as the final determined) motion detection result of the detection target corresponding to 14:01-14:02 can be obtained, which can be, for example, "94% in a forward rapidly accelerated motion state".

[0100] In this solution, by processing the motion trajectory of at least one key point on the detection target through multiple classifiers, the motion detection result of each key point can be obtained, and then the motion detection result of the detection target can be determined based on the motion detection result of each key point. Among them, the motion trajectory calculated based on the key point is less time-consuming, more accurate, and more consistent with the actual motion of the detection target than the motion trajectory calculated by optical flow, etc., and provides a more reliable input for subsequent motion detection; in addition, multiple motion classifiers utilize information such as pixel scale changes, real scale changes, and angle changes of the detection target. Compared with the method of using multi-frame optical flow to judge the motion state, it has higher robustness and the obtained motion detection result is more accurate. In addition, the motion state determined by the embodiment of the present application has a more specific and detailed feature description compared to simple motion states such as stillness and motion and simple speed and acceleration information. In summary, through the method of the embodiment of the present application, a more accurate motion detection result of the detection target can be obtained, thereby providing a more accurate adjustment basis for the adjustment module of the mobile station, which can better assist the driving of the mobile station.

[0101] To implement the various functions of the method provided in the embodiments of the present application, the execution subject of the method may include a hardware structure and a software module, and the aforementioned functions may be implemented in the form of a hardware structure, a software module, or a combination of a hardware structure and a software module. Certain of the aforementioned functions may be implemented in the form of a hardware structure, a software module, or a combination of a hardware structure and a software module.

[0102] 6 is a schematic diagram of the structure of a detection device provided in an embodiment of the present application. The detection device can be a device in a mobile station. The detection device 60 includes an acquisition unit 601, a first determination unit 602, a second determination unit 603, a processing unit 604, and a third determination unit 605:

[0103] The acquisition unit 601 is used to acquire multiple images, each of which includes a detection target. The operations performed by the acquisition unit 601 can be referred to the description of step S101 in FIG. 2 .

[0104] The first determining unit 602 is configured to determine at least one key point on the contour of the detection target. The operations performed by the first determining unit 602 can be referred to the description of step S102 in FIG. 2 .

[0105] The second determining unit 603 is used to determine the motion trajectory of each key point. The operations performed by the second determining unit 603 can be referred to the description of the contents of step S103 in FIG. 2 .

[0106] The processing unit 604 is configured to classify the motion trajectory of each key point using multiple classifiers to obtain a motion detection result for each key point. The multiple classifiers include: a stationary motion classifier, a non-parallel motion classifier, a high-speed parallel motion classifier, and a low-speed motion classifier; and the multiple classifiers also include one or more of the following: a motion angle classifier, a motion scale classifier, and a retrograde motion classifier. The operations performed by the processing unit 604 can be referred to in the description of step S104 in FIG. 2 .

[0107] The third determining unit 605 is configured to determine the motion detection result of the detection target based on the motion detection result of each key point. The operations performed by the third determining unit 605 can be referred to the description of step S105 in FIG. 2 .

[0108] In a possible implementation, the motion detection result includes a motion state and a first motion probability, where the first motion probability is used to indicate a probability of being in the motion state.

[0109] In one possible implementation, the classifier is used to determine a motion feature of the key point and a second motion probability of the key point being in the motion feature; the motion state is determined based on multiple motion features, and the first motion probability is determined based on multiple second motion probabilities.

[0110] In one possible implementation, the motion state includes one or more of the following: forward rapid motion state, forward slow motion state, stationary state, forward cut-in state, forward cut-out state, high-speed reverse motion state, low-speed reverse motion state, forward acceleration state, forward deceleration state, forward rapid acceleration state, forward rapid deceleration state, emergency stop state, crossing state, parallel motion state, lateral turning state, longitudinal motion state, U-turn state, and parameter information describing the motion state; wherein the parameter information describing the motion state includes one or more of the following: absolute speed information, relative speed information, and collision time information.

[0111] In one possible implementation, the motion features determined by the stationary motion classifier include: stationary and moving; the motion features determined by the non-parallel motion classifier include: lateral driving, cutting in, and cutting out; the motion features determined by the high-speed parallel motion classifier include: fast, high speed, acceleration, and rapid acceleration; the motion features determined by the low-speed motion classifier include: slow, deceleration, and rapid deceleration; the motion features determined by the motion angle classifier include: judging turning, U-turn, parallel driving, and forward / reverse motion; the motion features determined by the motion scale classifier include: forward / reverse acceleration; the motion features determined by the reverse motion classifier include: reverse driving and forward driving.

[0112] In one possible implementation, the motion detection result of the detection target corresponds to the first moment, and the third determination unit 605 is also used to: update the motion detection result of the detection target corresponding to the first moment based on the motion detection result of the detection target corresponding to the second moment and the motion detection result of the detection target, and the second moment is before the first moment.

[0113] In one possible implementation, the camera that acquires the multiple images is located on a mobile platform, and the acquisition unit 601 is further used to: acquire motion parameters of the mobile platform; the processing unit 604 is specifically used to: classify the motion trajectory of each key point through multiple classifiers based on the motion parameters of the mobile platform to obtain a motion detection result for each key point.

[0114] In one possible implementation, the first determination unit 602 is specifically used to: perform target detection on the area of ​​interest in the image to determine a bounding box containing the detected target; process the image in the bounding box to determine the outline of the detected target and the type of the detected target, which type includes one or more of the following: vehicle, pedestrian, building, roadblock; determine at least one key point on the outline of the detected target based on the outline and the type, and the position of the key point on the outline is determined based on the type.

[0115] In one possible implementation, the camera that acquires multiple images is located on a mobile platform, and the third determination unit 605 is further used to: determine at least one feature coordinate point in the bounding box; classify the motion trajectory of each feature coordinate point through a bounding box motion classifier to obtain the relative speed of the detection target relative to the mobile platform, and the third motion probability at the relative speed; and update the motion detection result of the detection target based on the relative speed and the third motion probability.

[0116] In one possible implementation, the multiple images include a first image at a third moment and a second image at a fourth moment, and the second determination unit 603 is specifically used to: determine the first position information of each key point in the first image; determine the second position information of each key point in the second image; and determine the motion trajectory of each key point based on the first position information and the second position information.

[0117] In a possible implementation, the multiple classifiers are connected in a cascade or average ensemble manner.

[0118] Specifically, the operations performed by the various units of the detection device 60 shown in FIG6 can refer to the relevant content of the method embodiment corresponding to FIG2 above, and will not be described in detail here. The above-mentioned various units can be implemented in the form of hardware, software, or a combination of hardware and software. In one embodiment, the functions of the various units in the above content can be implemented by one or more processors in the detection device 60.

[0119] 7 is a schematic diagram of the structure of another detection device provided in an embodiment of the present application. The detection device 70 can be used to implement the method described in the above method embodiment. For details, please refer to the description of the above method embodiment.

[0120] The detection device 70 may include one or more processors 701. The processor 701 may be a general-purpose processor or a dedicated processor, etc. The processor 701 may be used to control the detection device, execute software programs, and process data of the software programs.

[0121] Optionally, the detection device 70 may include one or more memories 702, on which program code 704 may be stored. The program code may be executed on the processor 701 so that the detection device 70 performs the method described in the above method embodiment. Optionally, data may also be stored in the memory 702. The processor 701 and memory 702 may be provided separately or integrated together. Optionally, the memory 702 may also be located outside the detection device 70 and coupled to the detection device 70 in some manner.

[0122] Optionally, the detection device 70 may further include a transceiver 705. The transceiver 705 may be referred to as a transceiver unit, a transceiver, or a transceiver circuit, and is configured to implement transceiver functions. The transceiver 705 may include a receiver and a transmitter. The receiver may be referred to as a receiver or a receiving circuit, and is configured to implement a receiving function; the transmitter may be referred to as a transmitter or a transmitting circuit, and is configured to implement a transmitting function.

[0123] The processor 701 is used to receive multiple images through the transceiver 705, where the images include a detection target; and is also used to determine at least one key point on the outline of the detection target; determine the motion trajectory of each key point; and classify the motion trajectory of each key point using multiple classifiers to obtain a motion detection result for each key point; wherein the multiple classifiers include: a stationary motion classifier, a non-parallel motion classifier, a high-speed parallel motion classifier, and a low-speed motion classifier; the multiple classifiers also include one or more of the following: a motion angle classifier, a motion scale classifier, a bounding box motion classifier, and a retrograde motion classifier; and determine the motion detection result of the detection target based on the motion detection result of each key point.

[0124] It should be noted that, in the above embodiment, the detection device 70 may be a device in a mobile station, or may be a chip, a chip system, or a processor that supports the implementation of the above method in the device in the mobile station.

[0125] Specifically, the operations performed by the detection device 70 may refer to the relevant contents about the detection module in the method embodiment corresponding to FIG. 2 , and will not be described in detail here.

[0126] In another possible design, the transceiver 705 may be a transceiver circuit, an interface, or an interface circuit. The transceiver circuit, interface, or interface circuit for implementing the receiving and transmitting functions may be separate or integrated. The transceiver circuit, interface, or interface circuit may be used for reading and writing code / data, or may be used for transmitting or delivering signals.

[0127] In another possible design, the processor 701 may optionally store program code 703. The program code 703 runs on the processor 701, enabling the detection device 70 to perform the method described in the above method embodiment. The program code 703 may be fixed in the processor 701. In this case, the processor 701 may be implemented by hardware.

[0128] In another possible design, the detection device 70 may include a circuit that can implement the sending, receiving, or communication functions in the aforementioned method embodiments.

[0129] The processor and transceiver described in this application can be implemented on an integrated circuit (IC), an analog IC, a radio frequency integrated circuit RFIC, a mixed signal IC, an application specific integrated circuit (ASIC), a printed circuit board (PCB), an electronic device, etc.

[0130] The detection device described in the above embodiments may be a network device or a terminal device, but the scope of the detection device described in this application is not limited thereto, and the structure of the detection device may not be limited to FIG7 . The detection device may be an independent device or may be part of a larger device. For example, the detection device may be:

[0131] (1) An independent integrated circuit IC, or chip, or chip system or subsystem; (2) A collection of one or more ICs, optionally, the IC collection may also include a storage component for storing data and program code; (3) ASIC, such as a modem; (4) A module that can be embedded in other devices; (5) Receivers, smart terminals, wireless devices, handheld devices, mobile units, vehicle-mounted devices, cloud devices, artificial intelligence devices, etc.

[0132] In the case where the detection device can be a chip or a chip system, please refer to the schematic diagram of the chip structure shown in Figure 8. The chip 80 shown in Figure 8 includes a logic circuit 801 and an input / output interface 802. The number of logic circuits 801 can be one or more, and the number of input / output interfaces 802 can be multiple.

[0133] The logic circuit 801 may be configured to acquire a plurality of images through the input and output interface 802 , wherein the images include detection targets.

[0134] The logic circuit 801 can also be used to determine at least one key point on the contour of the detection target; determine the motion trajectory of each key point; classify the motion trajectory of each key point through multiple classifiers to obtain a motion detection result for each key point; wherein the multiple classifiers include: a stationary motion classifier, a non-parallel motion classifier, a high-speed parallel motion classifier, and a low-speed motion classifier; the multiple classifiers also include one or more of the following: a motion angle classifier, a motion scale classifier, a bounding box motion classifier, and a retrograde motion classifier; and determine the motion detection result of the detection target based on the motion detection result of each key point.

[0135] Specifically, in this case, the operations performed by the logic circuit 801 may refer to the relevant contents of the method embodiment corresponding to FIG2 , and will not be described in detail here.

[0136] Those skilled in the art will also appreciate that the various illustrative logical blocks and steps listed in the embodiments of the present application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present application.

[0137] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer-readable storage medium is executed by a computer, the functions of any of the above method embodiments are realized.

[0138] The present application also provides a computer program product, which implements the functions of any of the above method embodiments when executed by a computer.

[0139] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a high-density digital video disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0140] Those skilled in the art will understand that the various numerical numbers such as first and second involved in this application are only for the convenience of description and are not used to limit the scope or sequence of the embodiments of this application.

[0141] The predefined in this application may be understood as defined, predefined, stored, pre-stored, pre-negotiated, pre-configured, solidified, or pre-burned.

[0142] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0143] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

Claims

1. A method for detecting a target, characterized in that: include: Acquire a plurality of images, wherein the images include a detection target; Determining at least one key point on the contour of the detection target; Determine the motion trajectory of each key point; Classify the motion trajectory of each key point using multiple classifiers to obtain a motion detection result for each key point; The multiple classifiers include: a stationary motion classifier, a non-parallel motion classifier, a high-speed parallel motion classifier, and a low-speed motion classifier; the multiple classifiers also include one or more of the following: a motion angle classifier, a motion scale classifier, and a retrograde motion classifier; The motion detection result of the detection target is determined based on the motion detection result of each key point.

2. The method according to claim 1, characterized in that The motion detection result includes a motion state and a first motion probability, where the first motion probability is used to indicate a probability of being in the motion state.

3. The method according to claim 2, characterized in that A classifier is used to determine a motion feature of the key point and a second motion probability of the key point being in the motion feature; The motion state is determined based on a plurality of the motion features, and the first motion probability is determined based on a plurality of the second motion probabilities.

4. The method according to claim 2 or 3, characterized in that The motion state includes one or more of the following: forward fast motion state, forward slow motion state, stationary state, forward cut-in state, forward cut-out state, high-speed reverse motion state, low-speed reverse motion state, forward accelerated motion state, forward decelerated motion state, forward rapid accelerated motion state, forward rapid decelerated motion state, emergency stop state, crossing state, parallel motion state, lateral turning state, longitudinal motion state, U-turn state, and parameter information describing the motion state; The parameter information describing the motion state includes one or more of the following: absolute speed information, relative speed information, and collision time information.

5. The method according to claim 3, characterized in that The motion features determined by the stationary-motion classifier include: stationary, moving; The motion features determined by the non-parallel motion classifier include: lateral travel, cut-in, and cut-out; The motion features determined by the high-speed parallel motion classifier include: fast, high speed, acceleration, and rapid acceleration; The motion features determined by the low-speed motion classifier include: slow speed, deceleration, and rapid deceleration; The motion features determined by the motion angle classifier include: judging turning, U-turn, parallel driving, and forward / reverse motion; The motion features determined by the motion scale classifier include: forward / reverse acceleration; The motion features determined by the reverse motion classifier include: reverse driving and forward driving.

6. The method according to any one of claims 1 to 5, characterized in that The motion detection result of the detection target corresponds to the first moment, and the method further includes: The motion detection result of the detection target corresponding to the first moment is updated based on the motion detection result of the detection target corresponding to a second moment and the motion detection result of the detection target, the second moment being before the first moment.

7. The method according to any one of claims 1 to 6, characterized in that The camera that obtains the multiple images is located on a mobile platform, and the motion trajectory of each key point is classified by multiple classifiers to obtain a motion detection result for each key point, including: Acquiring motion parameters of the mobile station; Based on the motion parameters of the mobile station, the motion trajectory of each key point is classified by multiple classifiers to obtain a motion detection result of each key point.

8. The method according to any one of claims 1 to 7, characterized in that Determining at least one key point on the contour of the detection target includes: Performing target detection on a region of interest in the image to determine a bounding box containing the detected target; Processing the image in the bounding box to determine the outline of the detection target and the type of the detection target, where the type includes one or more of the following: vehicle, pedestrian, building, and roadblock; At least one key point on the contour of the detection target is determined based on the contour and the type, and a position of the key point on the contour is determined based on the type.

9. The method according to claim 8, characterized in that The camera for acquiring the plurality of images is located on a mobile platform, and the method further comprises: Determining at least one feature coordinate point in the bounding box; Classifying the motion trajectory of each feature coordinate point using a bounding box motion classifier to obtain a relative speed of the detection target relative to the mobile station and a third motion probability of being at the relative speed; The motion detection result of the detection target is updated based on the relative speed and the third motion probability.

10. The method according to any one of claims 1 to 9, characterized in that The multiple images include a first image at a third moment and a second image at a fourth moment, and determining the motion trajectory of each key point includes: Determining first position information of each key point in the first image; Determining second position information of each key point in the second image; The motion trajectory of each key point is determined according to the first position information and the second position information.

11. The method according to any one of claims 1 to 10, characterized in that The multiple classifiers are connected in a cascade or average integration manner.

12. A detection device, characterized in that: The detection device includes an acquisition unit, a first determination unit, a second determination unit, a processing unit, and a third determination unit: The acquisition unit is configured to acquire a plurality of images, wherein the images include a detection target; The first determining unit is configured to determine at least one key point on the contour of the detection target; The second determining unit is used to determine the motion trajectory of each key point; The processing unit is configured to classify the motion trajectory of each key point using multiple classifiers to obtain a motion detection result for each key point; The multiple classifiers include: a stationary motion classifier, a non-parallel motion classifier, a high-speed parallel motion classifier, and a low-speed motion classifier; the multiple classifiers also include one or more of the following: a motion angle classifier, a motion scale classifier, a bounding box motion classifier, and a retrograde motion classifier; The third determining unit is configured to determine the motion detection result of the detection target based on the motion detection result of each key point.

13. The detection device according to claim 12, characterized in that: The motion detection result includes a motion state and a first motion probability, where the first motion probability is used to indicate a probability of being in the motion state.

14. The detection device according to claim 13, characterized in that: A classifier is used to determine a motion feature of the key point and a second motion probability of the key point being in the motion feature; The motion state is determined based on a plurality of the motion features, and the first motion probability is determined based on a plurality of the second motion probabilities.

15. The detection device according to claim 13 or 14, characterized in that: The motion state includes one or more of the following: forward fast motion state, forward slow motion state, stationary state, forward cut-in state, forward cut-out state, high-speed reverse motion state, low-speed reverse motion state, forward accelerated motion state, forward decelerated motion state, forward rapid accelerated motion state, forward rapid decelerated motion state, emergency stop state, crossing state, parallel motion state, lateral turning state, longitudinal motion state, U-turn state, and parameter information describing the motion state; The parameter information describing the motion state includes one or more of the following: absolute speed information, relative speed information, and collision time information.

16. The detection device according to claim 14, characterized in that The motion features determined by the stationary-motion classifier include: stationary, moving; The motion features determined by the non-parallel motion classifier include: lateral travel, cut-in, and cut-out; The motion features determined by the high-speed parallel motion classifier include: fast, high speed, acceleration, and rapid acceleration; The motion features determined by the low-speed motion classifier include: slow speed, deceleration, and rapid deceleration; The motion features determined by the motion angle classifier include: judging turning, U-turn, parallel driving, and forward / reverse motion; The motion features determined by the motion scale classifier include: forward / reverse acceleration; The motion features determined by the reverse motion classifier include: reverse driving and forward driving.

17. The detection device according to any one of claims 12 to 16, characterized in that: The motion detection result of the detection target corresponds to the first moment, and the third determining unit is further configured to: The motion detection result of the detection target corresponding to the first moment is updated based on the motion detection result of the detection target corresponding to a second moment and the motion detection result of the detection target, the second moment being before the first moment.

18. The detection device according to any one of claims 12 to 17, characterized in that: The camera for acquiring the plurality of images is located on a mobile platform, and the acquisition unit is further configured to: acquire motion parameters of the mobile platform; The processing unit is specifically configured to: perform classification processing on the motion trajectory of each key point through a plurality of classifiers based on the motion parameters of the mobile station, so as to obtain a motion detection result of each key point.

19. The detection device according to any one of claims 12 to 18, characterized in that: The first determining unit is specifically configured to: Performing target detection on a region of interest in the image to determine a bounding box containing the detected target; Processing the image in the bounding box to determine the outline of the detection target and the type of the detection target, where the type includes one or more of the following: vehicle, pedestrian, building, and roadblock; At least one key point on the contour of the detection target is determined based on the contour and the type, and a position of the key point on the contour is determined based on the type.

20. The detection device according to any one of claims 12 to 19, characterized in that: The camera for acquiring the multiple images is located on a mobile platform, and the third determining unit is further configured to: Determining at least one feature coordinate point in the bounding box; Classifying the motion trajectory of each feature coordinate point using a bounding box motion classifier to obtain a relative speed of the detection target relative to the mobile station and a third motion probability of being at the relative speed; The motion detection result of the detection target is updated based on the relative speed and the third motion probability.

21. The detection device according to any one of claims 12 to 20, characterized in that: The multiple images include a first image at a third moment and a second image at a fourth moment, and the second determining unit is specifically configured to: Determining first position information of each key point in the first image; Determining second position information of each key point in the second image; The motion trajectory of each key point is determined according to the first position information and the second position information.

22. The detection device according to any one of claims 12 to 21, characterized in that: The multiple classifiers are connected in a cascade or average integration manner.

23. A detection device, characterized in that: including processor and memory; The memory is used to store program code; The processor is configured to call the program code from the memory to execute the method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store instructions, and when the instructions are executed, the method according to any one of claims 1 to 11 is implemented.

Images