Motion state evaluation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of image processing, in particular to a motion state evaluation method and device, electronic equipment and a storage medium, which are used for reducing equipment cost and calculation complexity. The method comprises the following steps: loading video information, analyzing image information of a current frame, and obtaining an object and a current feature point in the current frame based on the image information; screening target current feature points which exist in the current frame and the previous frame at the same time from the current feature points; determining the motion state of the object based on the variable quantity of the direction angle of the vector between the current feature points of the two targets in the current frame and the direction angle of the corresponding vector in the previous frame; or, based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame, the motion state of the object is determined, and the historical feature point is the feature point contained in the previous frame. Therefore, the motion state evaluation of the object can be realized only depending on the video data, and the equipment cost and the calculation complexity are reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a motion state assessment method, device, electronic device and storage medium. Background Art

[0002] In the context of vigorously developing smart transportation, ensuring safe and efficient operation of traffic is the primary goal of smart transportation, and sensing the motion state of traffic participants is the key link to achieve this primary goal. Effective perception of the motion state of traffic participants can improve the accuracy of tracking trajectories of traffic participants and greatly reduce the complexity of judging various traffic events.

[0003] Currently, radar or point cloud technology is usually used to perceive the motion status of traffic participants, which leads to high equipment cost and high computational complexity. Summary of the invention

[0004] The embodiments of the present application provide a motion state assessment method, device, electronic device, and storage medium to reduce equipment cost and computational complexity.

[0005] The specific technical solutions provided by the embodiments of this application are as follows: In a first aspect, a motion state assessment method is provided, the method comprising: Load the video information, parse the image information of the current frame, and obtain the object and current feature points in the current frame based on the image information; From the current feature points, select the target current feature points that exist in both the current frame and the previous frame, and the target current feature points belong to the current feature points; Determine the motion state of the object based on the change in the direction angle of the vector between the current feature points of the two targets in the current frame and the direction angle of the corresponding vector in the previous frame; or, The motion state of the object is determined based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame, wherein the historical feature point is the feature point contained in the previous frame.

[0006] In a possible embodiment, if the target current feature point includes: a current background feature point and a current object feature point of the object, then based on the change in the direction angle of the vector between the two target current feature points in the current frame and the direction angle of the corresponding vector in the previous frame, determining the motion state of the object includes: Calculate the change in the direction angle of the vector from each current background feature point to each current object feature point in the current frame and the direction angle of the corresponding vector in the previous frame in the cache; Based on the obtained changes, the motion state of the object is determined.

[0007] Through the above method, the motion state of the object can be accurately evaluated through the change in the direction angle of the vector between the background feature point and the object feature point in the current frame and the previous frame.

[0008] In a possible embodiment, determining the motion state of the object based on each obtained change amount includes: Among the obtained changes, the total number of target changes whose values ​​are greater than the change threshold is calculated; If the ratio between the total number of target changes and the total number of vectors from current background feature points to current object feature points is greater than a first motion threshold, the motion state of the object is determined to be motion.

[0009] Through the above method, the motion state of the object is judged by the proportion, which further improves the accuracy of motion state assessment.

[0010] In a possible embodiment, after determining the motion state of the object based on each obtained change amount, the method further includes: When the object's motion state is moving, the direction angles of each vector corresponding to the object in the current frame are updated to the cache; When the motion state of the object is stationary, the direction angles of each vector corresponding to the object in the previous frame are used as the direction angles of each vector corresponding to the object in the current frame, and are updated to the cache.

[0011] Through the above method, it can be ensured that the object can be detected as moving even when the speed of the object is slow.

[0012] In a possible embodiment, if the target current feature point includes: the current object feature point of the object, then based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame, determining the motion state of the object includes: Randomly select N current object feature points from each current object feature point in the current frame, and calculate the displacement between the N current object feature points and the corresponding historical feature points in the previous frame in the cache; Based on the displacements corresponding to the N current object feature points, the motion state of the object is determined.

[0013] Through the above method, the motion state of the object can be quickly and accurately evaluated through the displacement between the object feature points in the current frame and the previous frame.

[0014] In a possible embodiment, determining the motion state of the object based on the displacements corresponding to N current object feature points includes: Based on the displacements corresponding to the N current object feature points, the N current object feature points are clustered to obtain various clusters, and the cluster with the largest total number of elements in each cluster is taken as the target cluster; If the ratio between the total number of elements in the target cluster and N is greater than the second motion threshold, the motion state of the object is determined to be motion.

[0015] Through the above method, the motion state of the object is judged by the consistency of displacement, which improves the accuracy of motion state assessment.

[0016] In a possible embodiment, the method further includes: Based on the image information, the object in the current frame and the current key point of the object are obtained, and the current key point is: a significant local position point in the object in the current frame; From the current key points, filter out the target current key points that exist in both the current frame and the previous frame; The motion state of the object is determined based on the displacement between the target current key point in the current frame and the corresponding historical key point in the previous frame, where the historical key point is: a significant local position point in the object in the previous frame.

[0017] Through the above method, the information of the key points is directly obtained. Through the displacement between the key points of the object in the current frame and the previous frame, the motion state of the object can be quickly and accurately evaluated, which is simpler and less complex.

[0018] In a possible embodiment, after obtaining the object and the current key point of the object in the current frame based on the image information, the method further includes: The current key point is input into the filter for filtering to obtain the filtered current key point.

[0019] Through the above method, it is possible to avoid camera shaking or unstable key point detection and positioning, which may lead to abnormal final evaluation results, thereby improving the accuracy of the evaluation.

[0020] In a second aspect, a motion state assessment device is provided, the device comprising: The acquisition module is used to load the video information, parse the image information of the current frame, and obtain the object and current feature points in the current frame based on the image information; A processing module is used to filter out target current feature points that exist in both the current frame and the previous frame from the current feature points, and the target current feature points belong to the current feature points; A first evaluation module is used to determine the motion state of the object based on the change in the direction angle of the vector between the current feature points of the two targets in the current frame and the direction angle of the corresponding vector in the previous frame; or, The motion state of the object is determined based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame, wherein the historical feature point is the feature point contained in the previous frame.

[0021] In a possible embodiment, if the target current feature point includes: a current background feature point and a current object feature point of the object, when determining the motion state of the object based on the change in the direction angle of the vector between the two target current feature points in the current frame and the direction angle of the corresponding vector in the previous frame, the first evaluation module is further used to: Calculate the change in the direction angle of the vector from each current background feature point to each current object feature point in the current frame and the direction angle of the corresponding vector in the previous frame in the cache; Based on the obtained changes, the motion state of the object is determined.

[0022] In a possible embodiment, when determining the motion state of the object based on each obtained change amount, the first evaluation module is further used to: Among the obtained changes, the total number of target changes whose values ​​are greater than the change threshold is calculated; If the ratio between the total number of target changes and the total number of vectors from current background feature points to current object feature points is greater than a first motion threshold, the motion state of the object is determined to be motion.

[0023] In a possible embodiment, after determining the motion state of the object based on each obtained change amount, the first evaluation module is further configured to: When the object's motion state is moving, the direction angles of each vector corresponding to the object in the current frame are updated to the cache; When the motion state of the object is stationary, the direction angles of each vector corresponding to the object in the previous frame are used as the direction angles of each vector corresponding to the object in the current frame, and are updated to the cache.

[0024] In a possible embodiment, if the target current feature point includes: a current object feature point of the object, then when determining the motion state of the object based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame, the first evaluation module is further used to: Randomly select N current object feature points from each current object feature point in the current frame, and calculate the displacement between the N current object feature points and the corresponding historical feature points in the previous frame in the cache; Based on the displacements corresponding to the N current object feature points, the motion state of the object is determined.

[0025] In a possible embodiment, when determining the motion state of the object based on the displacements corresponding to the N current object feature points, the first evaluation module is further configured to: Based on the displacements corresponding to the N current object feature points, the N current object feature points are clustered to obtain various clusters, and the cluster with the largest total number of elements in each cluster is taken as the target cluster; If the ratio between the total number of elements in the target cluster and N is greater than the second motion threshold, the motion state of the object is determined to be motion.

[0026] In a possible embodiment, the device further includes a second evaluation module, and the second evaluation module is used to: Based on the image information, the object in the current frame and the current key point of the object are obtained, and the current key point is: a significant local position point in the object in the current frame; From the current key points, filter out the target current key points that exist in both the current frame and the previous frame; The motion state of the object is determined based on the displacement between the target current key point in the current frame and the corresponding historical key point in the previous frame, where the historical key point is: a significant local position point in the object in the previous frame.

[0027] In a possible embodiment, after obtaining the object and the current key point of the object in the current frame based on the image information, the second evaluation module is further used to: The current key point is input into the filter for filtering to obtain the filtered current key point.

[0028] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods described in the first aspect are implemented.

[0029] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0030] In the embodiment of the present application, the video information is first loaded, the image information of the current frame is parsed, and based on the image information, the object and the current feature point in the current frame are obtained, and then the target current feature points that exist in both the current frame and the previous frame are screened out from the current feature points, and the target current feature points belong to the current feature points. Finally, the motion state of the object is determined based on the direction angle of the vector between the two target current feature points in the current frame and the change in the direction angle of the corresponding vector in the previous frame; or, the motion state of the object is determined based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame, wherein the historical feature point is the feature point contained in the previous frame. In this way, the motion state evaluation of the object can be realized by relying only on the video data collected by the shooting device, which has obvious advantages in equipment cost and reduces equipment cost. In addition, the motion state of the object is determined based on the change in the vector direction angle between the current frame and the previous frame, or based on the displacement of the feature points in the current frame and the previous frame, which ensures the accuracy of the evaluation while reducing the calculation complexity, and the adaptability of the computing power equipment is strong. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of possible application scenarios in the embodiments of the present application; Figure 2 A schematic diagram of a first process flow of a motion state assessment method provided in an embodiment of the present application; Figure 3 A second flow chart of a motion state assessment method provided in an embodiment of the present application; Figure 4 A third flow chart of a motion state assessment method provided in an embodiment of the present application; Figure 5 A fourth flow chart of a motion state assessment method provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of a motion state assessment device provided in an embodiment of the present application; Figure 7 Schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0033] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0034] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0035] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0036] The following is a brief introduction to the design concept of the embodiment of the present application: In the context of vigorously developing smart transportation, ensuring safe and efficient operation of traffic is the primary goal of smart transportation, and sensing the motion state of traffic participants is the key link to achieve this primary goal. Effective perception of the motion state of traffic participants can improve the accuracy of tracking trajectories of traffic participants and greatly reduce the complexity of judging various traffic events.

[0037] Currently, radar or point cloud technology is usually used to perceive the motion status of traffic participants, which leads to high equipment cost and high computational complexity.

[0038] Therefore, a simple and efficient motion state assessment method plays a vital role in traffic accident warning and situational handling, vehicle inspection-related links, traffic management and planning optimization, etc.

[0039] In view of this, in an embodiment of the present application, a motion state assessment method, device, electronic device and storage medium are provided, which first load video information, parse the image information of the current frame, and obtain the object and current feature points in the current frame based on the image information. Then, from the current feature points, the target current feature points that exist in both the current frame and the previous frame are screened out, and the target current feature points belong to the current feature points. Finally, based on the direction angle of the vector between the two target current feature points in the current frame and the change in the direction angle of the corresponding vector in the previous frame, the motion state of the object is determined; or, based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame, the motion state of the object is determined, wherein the historical feature point is the feature point contained in the previous frame. In this way, the motion state of the object can be evaluated by relying only on the video data collected by the shooting equipment, which has obvious advantages in equipment cost and reduces the equipment cost. In addition, the motion state of the object can be determined based on the change between the direction angle of the vector in the current frame and the direction angle of the vector in the previous frame, or based on the displacement between the current feature point of the target in the current frame and the corresponding historical feature point in the previous frame. While ensuring the accuracy of the evaluation, the calculation complexity is reduced, and the computing power equipment has strong adaptability.

[0040] The preferred implementation modes of the present application are described in detail below with reference to the accompanying drawings.

[0041] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. It is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes a processing device 110 and a shooting device 120. The shooting device 120 is used to collect video data in a target tracking scenario. The processing device 110 receives the video data sent by the shooting device 120, and evaluates the motion state of the object based on the video data.

[0042] The processing device 110 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The shooting device 120 and the processing device 110 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0043] Based on the above embodiments, see Figure 2 FIG. 1 is a schematic diagram of a first process of a motion state assessment method in an embodiment of the present application, which specifically includes: Step 20: Load the video information, parse the image information of the current frame, and obtain the object and current feature points in the current frame based on the image information.

[0044] Among them, the objects are various vehicles, pedestrians and other traffic participants.

[0045] In an embodiment of the present application, video information is loaded, image information of the current frame is parsed, the parsed image information is sent to a full-image feature point calculation module to obtain the current feature point, and the parsed image information is sent to an object detection module to obtain the object in the current frame and its coordinate information.

[0046] In addition, it is worth noting that in the embodiments of the present application, the methods used by the full-image feature point calculation module include but are not limited to: traditional methods (such as scale-invariant feature transform (SIFT), accelerated robust features (SURF) and its derivative algorithms, etc.), and deep learning methods (such as deep matcher (LightGlue), etc. and its deformation network, etc.). The object detection module includes but is not limited to YOLO and its derivative networks.

[0047] Step 21: From the current feature points, filter out the target current feature points that exist in both the current frame and the previous frame.

[0048] In the embodiment of the present application, co-visibility feature matching is adopted to filter out target current feature points that exist in both the current frame and the previous frame from the current feature points, that is, feature points that are commonly visible in the frames in the two figures.

[0049] In this way, the feature points with different viewing angles but the same features in the current frame and the previous frame are matched, and the feature points that exist in both the current frame and the previous frame are accurately found.

[0050] Optionally, in an embodiment of the present application, the object in the current frame is tracked to obtain a tracking index of the object in the current frame, and then the object is segmented to obtain a mask of the object, and finally, based on the mask information of the object, the current feature points of the target are classified into current background feature points and current object feature points of the object.

[0051] Among them, the objects can be segmented using the segmentation model or other semantic segmentation models to obtain independent masks of the objects.

[0052] Specifically, in the embodiment of the present application, the target current feature point within the mask of the object is used as the current object feature point of the object. pi j , the remaining target current feature points outside the object mask are used as current background feature points b k .

[0053] in, i is the tracking index of the object, j is the index of a current feature point of the object, k It is the index of the current background feature point.

[0054] Step 22: Determine the motion state of the object based on the change in the direction angle of the vector between the two target current feature points in the current frame and the direction angle of the corresponding vector in the previous frame; or determine the motion state of the object based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame.

[0055] The historical feature points are feature points contained in the previous frame, the motion state is divided into motion and stillness, and the orientation change includes the change of the azimuth angle or the change of the displacement.

[0056] In the embodiment of the present application, two possible implementations are provided to determine the motion state of the target object.

[0057] The first method: The current feature points of the target include: the current background feature points and the current object feature points of the object. The motion state of the object is determined based on the change in the direction angle of the vector between the two current feature points of the target in the current frame and the direction angle of the corresponding vector in the previous frame. Specifically, it includes: Step A1: Calculate the change between the direction angle of the vector pointing from each current background feature point to each current object feature point in the current frame and the direction angle of the corresponding vector in the previous frame in the cache.

[0058] Among them, the change in the direction angle of a vector can be expressed as: , The current frame k The background feature points point to i The object's j The direction angle of the vector of the object feature point, The previous frame k The background feature points point to i The object's j The direction angle of the vector of the object feature point, f is the frame number of the current frame, i is the tracking index of the object, j is the index of an object feature point of the object, k The index of a background feature point.

[0059] In addition, it is worth noting that in the embodiment of the present application, a random selection of the current background feature points can be performed. m The current background feature points are randomly selected from the current object feature points of the object. n The feature points of the current object are then m The current background feature point is the starting point. n The feature points of the current object are connected as the end points to form a vector , the total number of vectors is m × n , m is an integer greater than 1 and less than or equal to the total number of current background feature points. n is an integer greater than 1 and less than or equal to the total number of feature points of the current object. m The current background feature points point to the object n The change between the direction angle of the vector of the current object feature point and the direction angle of the corresponding vector in the previous frame in the cache .

[0060] In this way, a certain number of current background feature points and current object feature points are randomly selected to calculate the change in the direction angle of the vector, which can reduce the amount of calculation and save computing resources.

[0061] In addition, it is worth noting that in the embodiment of the present application, the change in the direction angle of the vector pointing to each current background feature point from each current object feature point in the current frame and the direction angle of the corresponding vector in the previous frame in the cache can also be calculated. The embodiment of the present application does not restrict the direction of the vector.

[0062] Step A2: Determine the motion state of the target object based on the obtained changes.

[0063] The obtained changes can be the changes in the direction angles of the vectors of all the current background feature points pointing to all the current object feature points in the current frame and the direction angles of the corresponding vectors in the previous frame in the cache, or the changes in the direction angles of the vectors of all the current background feature points pointing to all the current object feature points in the current frame. m The current background feature points point to the object n The change in the direction angle of the vector of the current object feature point and the direction angle of the corresponding vector in the previous frame in the cache is not limited in the embodiment of the present application.

[0064] In this way, the motion state of the object can be accurately evaluated through the change in the direction angle of the vector between the background feature point and the object feature point in the current frame and the previous frame.

[0065] Optionally, in the embodiment of the present application, a possible embodiment is provided to determine the motion state of the target object based on each obtained change amount, and the specific steps are as follows: Step A21: Count the total number of target changes whose values ​​are greater than the change threshold among the obtained changes.

[0066] In the embodiment of the present application, the changes in the direction angles of each vector are traversed. If the value of the change in the direction angle of a vector is greater than the change threshold, the change is used as the target change, and the total number of target changes is added by 1.

[0067] The change threshold is a threshold for determining whether a vector has changed.

[0068] Step A22: Determine whether the ratio between the total number of target changes and the total number of vectors from the current background feature points to the current object feature points is greater than the first motion threshold. If so, execute step A23; otherwise, execute step A24.

[0069] In the embodiment of the present application, the ratio between the total number of target changes and the total number of vectors from each current background feature point to each current object feature point is T i / S ,in, T i For object i The total number of corresponding target changes, S Point to the object for each current background feature point i The total number of vectors of the feature points of the current object, that is, the total number of changes obtained, and the changes obtained are the total number of changes in the current frame. m The current background feature points point to the object n When the direction angle of the vector of the current object feature point changes with the direction angle of the corresponding vector in the previous frame in the cache, S=m × n .

[0070] In the embodiment of the present application, the first motion threshold is a threshold for determining whether the position of the object has changed, and is a threshold for the ratio of the total number of target changes to the total number of changes.

[0071] Step A23: Determine that the motion state of the object is motion.

[0072] In the embodiment of the present application, if the ratio between the total number of target changes and the total number of vectors from the current background feature points to the current object feature points is greater than the first motion threshold, the motion state of the object is determined to be motion.

[0073] Step A24: Determine that the motion state of the object is stationary.

[0074] In the embodiment of the present application, if the ratio between the total number of target changes and the total number of vectors from the current background feature points to the current object feature points is less than or equal to the first motion threshold, the motion state of the object is determined to be stationary.

[0075] In this way, the motion state of the object is judged by the proportion, which further improves the accuracy of motion state assessment.

[0076] In addition, it is worth noting that in the embodiment of the present application, it can also be determined whether the total number of target changes is greater than the third motion threshold. If the total number of target changes is greater than the third motion threshold, the motion state of the object is determined to be motion. This is not limited in the embodiment of the present application.

[0077] The third motion threshold is determined based on the total number of changes obtained and the first motion threshold. The third motion threshold is a threshold for determining whether the position of the object has changed, and is a threshold for the total number of target changes.

[0078] Further, in the embodiment of the present application, after determining the motion state of the object based on the obtained changes, the direction angles of the vectors corresponding to the object in the current frame are cached, including the following two cases: Case 1: When the motion state of the object is moving, the direction angles of each vector corresponding to the object in the current frame are updated to the cache.

[0079] Exemplarily, the direction angles of the vectors of the current frame in the cache are .

[0080] Case 2: When the object is in a stationary state, the direction angles of the vectors corresponding to the object in the previous frame are used as the direction angles of the vectors corresponding to the object in the current frame, and are updated to the cache.

[0081] Exemplarily, the direction angles of the vectors of the current frame in the cache are .

[0082] In this way, since the change in the direction angle between the current frame and the previous frame is small, the direction angle of the previous frame is used as the direction angle of the current frame, so that when the next frame is compared with the current frame, the change in the direction angle is obvious, which can ensure that the target object can be detected as moving even when the speed of the target object is slow.

[0083] The second method: the target current feature point includes: the current object feature point of the object, then the motion state of the object is determined based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame, specifically including: Step B1: Randomly select N current object feature points from each current object feature point in the current frame, and calculate the displacement between the N current object feature points and the corresponding historical feature points in the previous frame in the cache.

[0084] Among them, the displacement corresponding to a feature point of the current object can be expressed as: , For the object in the current frame i No. j The location of the feature points of the object, For the object in the previous frame i No. j The location of the feature points of the object, f is the frame number of the current frame, i is the tracking index of the object, j The index of an object feature point of the object.

[0085] In the embodiment of the present application, N current object feature points that exist in both the current frame and the previous frame are randomly selected from the current object feature points in the current frame, where N is an integer greater than 1 and less than or equal to the total number of current object feature points. The displacements of the N current object feature points in the current frame and the corresponding historical feature points in the previous frame in the cache are calculated.

[0086] In this way, randomly selecting a certain number of feature points of each current object and calculating the displacement can reduce the amount of calculation and save computing resources.

[0087] Step B2: Determine the motion state of the object based on the displacements corresponding to the N current object feature points.

[0088] In this way, the motion state of the object can be quickly and accurately evaluated through the displacement between the feature points of the object in the current frame and the previous frame.

[0089] Optionally, in an embodiment of the present application, a possible embodiment is provided to determine the motion state of an object based on the displacements corresponding to N current object feature points. The specific steps are as follows: Step B21: clustering the N current object feature points based on the displacements corresponding to the N current object feature points to obtain various clusters, and taking the cluster with the largest total number of elements in each cluster as the target cluster.

[0090] In an embodiment of the present application, the displacement similarity between each pair of current object feature points is calculated, and the N current object feature points are clustered according to the displacement similarity to obtain various clusters, and then the total number of elements within each cluster is counted, and the cluster with the largest total number of elements within each cluster is used as the target cluster.

[0091] Among them, clustering methods include but are not limited to K-means and HDBSCAN clustering methods. Finally, the clustering results are analyzed to count the total number of elements within each cluster. The total number of elements within a cluster is the total number of current object feature points contained in the cluster.

[0092] For example, the displacement similarity may be cosine similarity, and the displacement similarity between any two current object feature points may be expressed as: ,in, a and b All objects i The index of the feature point of the current object in , and a Not equal to b , and Respectively i The current object feature point a With the current object feature point b The corresponding displacement.

[0093] Step B22: Determine whether the ratio between the total number of elements in the target cluster and N is greater than the second motion threshold. If so, execute step B23; otherwise, execute step B24.

[0094] In the embodiment of the present application, the ratio between the total number of elements in each target cluster and N is FC i / N ,in, FC i For object i The total number of elements in the corresponding target cluster, N For object i The total number of feature points of the corresponding selected current object.

[0095] In the embodiment of the present application, the second motion threshold is a threshold for determining whether the position of the object has changed, and is a threshold for the ratio of the total number of elements in the target cluster to the total number of feature points selected for the current object.

[0096] Step B23: Determine that the motion state of the object is motion.

[0097] In the embodiment of the present application, if the ratio between the total number of elements in the target cluster and N is greater than the second motion threshold, the motion state of the object is determined to be motion.

[0098] Step B24: Determine that the motion state of the object is stationary.

[0099] In the embodiment of the present application, if the ratio between the total number of elements in the target cluster and N is not greater than the second motion threshold, it is determined that the motion state of the object is stationary.

[0100] In this way, the motion state of the object is judged by the consistency of displacement, which improves the accuracy of motion state assessment.

[0101] Further, in the embodiment of the present application, after determining the motion state of the object based on the obtained displacements, information of the object feature points corresponding to the object in the current frame is cached, including the following two cases: Case 1: When the motion state of the object is moving, the information of the current object feature point of the object in the current frame is updated to the cache.

[0102] The information of the feature point of the current object includes but is not limited to the position information of the feature point of the current object.

[0103] Case 2: When the motion state of the object is stationary, the information of the historical object feature points of the object in the previous frame is used as the information of the current object feature points of the object in the current frame, and is updated to the cache.

[0104] In this way, since the displacement of the object feature points between the current frame and the previous frame is small at this time, the information of the object feature points of the previous frame is used as the information of the object feature points of the current frame, so that when the next frame is compared with the current frame, the displacement of the object feature points is smaller, which can ensure that the target object can be detected as moving even when the object speed is slow.

[0105] In addition, it is worth noting that in the embodiments of the present application, the above two methods of determining the motion state of an object can be combined to comprehensively determine the motion state of the object. For example, the first method determines that the motion state of the object is motion, and the second method determines that the motion state of the object is stationary, then the motion state of the object is finally determined to be motion.

[0106] Optionally, in the embodiment of the present application, after the video information is loaded and the image information of the current frame is parsed, another motion state evaluation method may be used based on the image information, see Figure 3 FIG. 1 is a second flow chart of a motion state assessment method in an embodiment of the present application, which specifically includes: Step 30: Based on the image information, obtain the object in the current frame and the current key point of the object.

[0107] The current key point is a significant local position point in the object in the current frame.

[0108] In the embodiment of the present application, based on the image information, the object and the current key point in the current frame are detected through a detection network.

[0109] When obtaining the current key point, the detected object is tracked to obtain the current key point in the object, wherein i is the tracking index of the object, jThe index of the current key point.

[0110] The detection network includes but is not limited to methods such as YOLO that can detect objects. The current key points of the object can be pre-marked. Taking a vehicle as an example, the local position points that are easy to distinguish can be pre-marked, such as the headlights, license plates, rearview mirrors, four corner points of the windshield, two end points of the wipers, and the location where the tires touch.

[0111] Furthermore, based on the image information, after obtaining the object in the current frame and the current key point of the object, the current key point can be input into the filter for filtering to obtain the coordinates of the filtered current key point. ,in i is the tracking index of the object, j is the index of the current key point, f The frame number of the current frame.

[0112] The filter includes but is not limited to the Kalman filter.

[0113] In this way, it is possible to avoid camera shake or unstable key point detection and positioning, which may lead to abnormal final evaluation results and improve the accuracy of the evaluation.

[0114] Step 31: From the current key points, filter out the target current key points that exist in both the current frame and the previous frame.

[0115] In the embodiment of the present application, common view feature matching is adopted to filter out the target current key points that exist simultaneously in the current frame and the previous frame, that is, the key points that are visible to both image frames.

[0116] For example, the object i The number of key points that exist simultaneously in the previous and next frames is m i .

[0117] In this way, the key points with different perspectives but the same features in the current frame and the previous frame are matched, and the key points that exist in both the current frame and the previous frame are accurately found.

[0118] Step 32: Determine the motion state of the object based on the displacement between the target current key point in the current frame and the corresponding historical key point in the previous frame.

[0119] Among them, the historical key points are: the significant local position points in the object in the previous frame.

[0120] In an embodiment of the present application, M target current key points are randomly selected from the target current key points, and the displacement of the M target current key points and the corresponding historical key points in the previous frame in the cache is calculated, and the motion state of the object is determined based on the displacement corresponding to the M target current key points.

[0121] Wherein, M is an integer greater than 1 and less than or equal to the total number of current key points of the target of the object.

[0122] In an embodiment of the present application, M target current key points that exist simultaneously in the current frame and the previous frame are randomly selected from the target current key points of the object in the current frame, and the displacement of the M target current key points in the current frame and the corresponding historical key points in the previous frame in the cache is calculated, and then the displacement similarity between each two of the M target current key points is calculated, and the M target current key points are clustered according to the displacement similarity to obtain various clusters, and then the total number of elements within each cluster is counted, and the cluster with the largest total number of elements within each cluster is used as the target cluster. If the ratio between the total number of elements within the target cluster and M is greater than the fourth motion threshold, the motion state of the object is determined to be motion, otherwise, the motion state of the object is determined to be stationary.

[0123] Among them, the fourth motion threshold is the threshold for judging whether the position of the object has changed. It is the threshold of the ratio of the total number of elements within the target cluster to the total number of current key points of the selected target. The clustering methods include but are not limited to clustering methods such as K-means and HDBSCAN. Finally, the clustering results are analyzed to count the total number of elements within each cluster. The total number of elements within a cluster is the total number of current key points of the target contained in the cluster.

[0124] The displacement corresponding to the current key point of a target can be expressed as: , For the object in the current frame i No. j The location of the key points, For the object in the previous frame i No. j The location of the key points, f is the frame number of the current frame and is an integer greater than or equal to 3, i is the tracking index of an object, j The index of a key point of the object.

[0125] The displacement similarity between any two current key points of the target can be expressed as: ,in, p and q All objects i The index of the current key point of the target in p Not equal to q , and Respectively i Current key point of target p Current key point with target q The corresponding displacement.

[0126] In this way, the information of the key points is obtained directly, and the motion state of the object can be quickly and accurately evaluated through the displacement between the key points of the object in the current frame and the previous frame. Compared with the method of determining the motion state of the object based on the change in the direction angle of the vector pointing to the current object feature points of the object based on the current background feature points in the current frame and the direction angle of the corresponding vector in the previous frame in the cache, this method is simpler and less complex.

[0127] Based on the above embodiments, see Figure 4 FIG. 1 is a schematic diagram of a third process of a motion state assessment method in an embodiment of the present application, which specifically includes: Step 40: Load the video information and parse the image information of the current frame.

[0128] Step 41: Based on the image information, obtain the object and current feature points in the current frame.

[0129] Wherein, obtaining the object in the current frame includes object detection, object tracking and object segmentation.

[0130] Step 42: Filter out the target current feature points that exist in both the current frame and the previous frame from the current feature points.

[0131] Step 43: Classify the current feature points of the target to obtain the current background feature points and the current object feature points of the object.

[0132] Step 44: Calculate the change between the direction angle of the vector pointing from each current background feature point in the current frame to each current object feature point of the object and the direction angle of the corresponding vector in the previous frame in the cache.

[0133] Step 45: Count the total number of target changes whose values ​​are greater than the change threshold among the obtained changes.

[0134] Step 46: Determine whether the ratio between the total number of target changes and the total number of vectors of each current object feature point pointing to the object from each current background feature point is greater than the first motion threshold; if so, execute step 47; otherwise, execute step 49.

[0135] Step 47: Determine that the motion state of the target object is motion.

[0136] Step 48: Update the direction angles of each vector corresponding to the object in the current frame into the cache.

[0137] Step 49: Determine that the motion state of the target object is stationary.

[0138] Step 410: The direction angles of each vector corresponding to the object in the previous frame are used as the direction angles of each vector corresponding to the object in the current frame, and updated into the cache.

[0139] Based on the above embodiments, see Figure 5 FIG. 4 is a schematic diagram of a fourth process of a motion state assessment method in an embodiment of the present application, which specifically includes: Step 50: Load the video information and parse the image information of the current frame.

[0140] Step 51: Based on the image information, obtain the object in the current frame and the current key point of the object.

[0141] The current key point is a significant local position point in the object in the current frame.

[0142] Step 52: Input the current key point into the filter for filtering to obtain the filtered current key point.

[0143] Step 53: From the current key points, filter out the target current key points that exist in both the current frame and the previous frame.

[0144] Step 54: Randomly select M target current key points from the target current key points, and calculate the displacement of the M target current key points and the corresponding historical key points in the previous frame in the cache.

[0145] Among them, the historical key points are: the significant local position points in the object in the previous frame.

[0146] Step 55: Based on the displacements corresponding to the current key points of the M targets, cluster the current key points of the M targets to obtain various clusters.

[0147] Step 56: The cluster with the largest total number of elements in each cluster is taken as the target cluster.

[0148] Step 57: Determine whether the ratio between the total number of elements in the target cluster and M is greater than the fourth motion threshold; if so, execute step 58; otherwise, execute step 510.

[0149] Step 58: Determine that the motion state of the object is motion.

[0150] Step 59: Update the information of the target current key point of the object in the current frame into the cache.

[0151] Step 510: Determine that the motion state of the object is stationary.

[0152] Step 511: Use the historical key point information of the object in the previous frame as the target current key point information of the object in the current frame, and update it into the cache.

[0153] Based on the same inventive concept, the present application embodiment provides a motion state evaluation device, please refer to Figure 6 , the device comprises: The acquisition module 601 is used to load the video information, parse the image information of the current frame, and obtain the object and the current feature point in the current frame based on the image information; Processing module 602, used to filter out target current feature points that exist in both the current frame and the previous frame from the current feature points, and the target current feature points belong to the current feature points; The first evaluation module 603 is used to determine the motion state of the object based on the change in the direction angle of the vector between the two current feature points of the target in the current frame and the direction angle of the corresponding vector in the previous frame; or The motion state of the object is determined based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame, wherein the historical feature point is the feature point contained in the previous frame.

[0154] In a possible embodiment, if the target current feature point includes: a current background feature point and a current object feature point of the object, when determining the motion state of the object based on the change in the direction angle of the vector between the two target current feature points in the current frame and the direction angle of the corresponding vector in the previous frame, the first evaluation module 603 is further used to: Calculate the change in the direction angle of the vector from each current background feature point to each current object feature point in the current frame and the direction angle of the corresponding vector in the previous frame in the cache; Based on the obtained changes, the motion state of the object is determined.

[0155] In a possible embodiment, when determining the motion state of the object based on each obtained change amount, the first evaluation module 603 is further configured to: Among the obtained changes, the total number of target changes whose values ​​are greater than the change threshold is calculated; If the ratio between the total number of target changes and the total number of vectors from current background feature points to current object feature points is greater than a first motion threshold, the motion state of the object is determined to be motion.

[0156] In a possible embodiment, after determining the motion state of the object based on each obtained change amount, the first evaluation module 603 is further configured to: When the object's motion state is moving, the direction angles of each vector corresponding to the object in the current frame are updated to the cache; When the motion state of the object is stationary, the direction angles of each vector corresponding to the object in the previous frame are used as the direction angles of each vector corresponding to the object in the current frame, and are updated to the cache.

[0157] In a possible embodiment, if the target current feature point includes: the current object feature point of the object, then when determining the motion state of the object based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame, the first evaluation module 603 is further used to: Randomly select N current object feature points from each current object feature point in the current frame, and calculate the displacement between the N current object feature points and the corresponding historical feature points in the previous frame in the cache; Based on the displacements corresponding to the N current object feature points, the motion state of the object is determined.

[0158] In a possible embodiment, when determining the motion state of the object based on the displacements corresponding to the N current object feature points, the first evaluation module 603 is further configured to: Based on the displacements corresponding to the N current object feature points, the N current object feature points are clustered to obtain various clusters, and the cluster with the largest total number of elements in each cluster is taken as the target cluster; If the ratio between the total number of elements in the target cluster and N is greater than the second motion threshold, the motion state of the object is determined to be motion.

[0159] In a possible embodiment, the apparatus further includes a second evaluation module 604, and the second evaluation module 604 is configured to: Based on the image information, the object in the current frame and the current key point of the object are obtained, and the current key point is: a significant local position point in the object in the current frame; From the current key points, filter out the target current key points that exist in both the current frame and the previous frame; The motion state of the object is determined based on the displacement between the target current key point in the current frame and the corresponding historical key point in the previous frame, where the historical key point is: a significant local position point in the object in the previous frame.

[0160] In a possible embodiment, after obtaining the object and the current key point of the object in the current frame based on the image information, the second evaluation module 604 is further configured to: The current key point is input into the filter for filtering to obtain the filtered current key point.

[0161] Based on the above embodiments, see Figure 7 Shown is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0162] An embodiment of the present application provides an electronic device, which may include a processor 710 (Center Processing Unit, CPU), a memory 720, an input device 730 and an output device 740, etc. The input device 730 may include a keyboard, a mouse, a touch screen, etc., and the output device 740 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0163] The memory 720 may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor 710 with program instructions and data stored in the memory 720. In the embodiment of the present application, the memory 720 may be used to store the program of any motion state assessment method in the embodiment of the present application.

[0164] The processor 710 calls the program instructions stored in the memory 720, and the processor 710 is used to execute any one of the motion state assessment methods in the embodiments of the present application according to the obtained program instructions.

[0165] Based on the above embodiments, in an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the motion state assessment method in any of the above method embodiments is implemented.

[0166] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0167] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0168] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0170] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for evaluating a motion state, characterized in that: include: Loading video information, parsing image information of the current frame, and obtaining objects and current feature points in the current frame based on the image information; Filtering out target current feature points that exist simultaneously in the current frame and the previous frame from the current feature points, the target current feature points belonging to the current feature points; Determining the motion state of the object based on a change in a direction angle of a vector between two current feature points of the target in the current frame and a direction angle of a corresponding vector in the previous frame; or, The motion state of the object is determined based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame, wherein the historical feature point is a feature point included in the previous frame.

2. The method according to claim 1, characterized in that If the target current feature point includes: a current background feature point and a current object feature point of the object, then determining the motion state of the object based on a change in a direction angle of a vector between two target current feature points in the current frame and a direction angle of a corresponding vector in the previous frame includes: Calculate the change in the direction angle of the vector from each current background feature point in the current frame to each current object feature point and the direction angle of the corresponding vector in the previous frame in the cache; Based on the obtained changes, the motion state of the object is determined.

3. The method according to claim 2, characterized in that The step of determining the motion state of the object based on each obtained change amount includes: Among the obtained changes, the total number of target changes whose values ​​are greater than the change threshold is calculated; If the ratio between the total number of target changes and the total number of vectors from the current background feature points to the current object feature points is greater than a first motion threshold, the motion state of the object is determined to be motion.

4. The method according to claim 2, characterized in that After determining the motion state of the object based on the obtained changes, the method further includes: When the motion state of the object is motion, updating the direction angles of each vector corresponding to the object in the current frame into the cache; When the motion state of the object is stationary, the direction angles of each vector corresponding to the object in the previous frame are used as the direction angles of each vector corresponding to the object in the current frame, and updated to the cache.

5. The method according to claim 1, characterized in that If the target current feature point includes: the current object feature point of the object, then determining the motion state of the object based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame includes: Randomly selecting N current object feature points from each current object feature point in the current frame, and calculating the displacement between the N current object feature points and the corresponding historical feature points in the previous frame in the cache; The motion state of the object is determined based on the displacements corresponding to the N current object feature points.

6. The method according to claim 5, characterized in that The determining the motion state of the object based on the displacements corresponding to the N current object feature points includes: Based on the displacements corresponding to the N current object feature points, the N current object feature points are clustered to obtain various clusters, and the cluster with the largest total number of elements in each cluster is taken as the target cluster; If the ratio of the total number of elements in the target cluster to N is greater than a second motion threshold, the motion state of the object is determined to be motion.

7. The method according to claim 1, characterized in that The method further comprises: Based on the image information, the object in the current frame and the current key point of the object are obtained, wherein the current key point is a significant local position point of the object in the current frame; Filtering out target current key points that exist simultaneously in the current frame and the previous frame from the current key points; The motion state of the object is determined based on the displacement between the target current key point in the current frame and the corresponding historical key point in the previous frame, wherein the historical key point is a significant local position point in the object in the previous frame.

8. The method according to claim 7, characterized in that After obtaining the object and the current key point of the object in the current frame based on the image information, the method further includes: The current key point is input into a filter for filtering to obtain the filtered current key point.

9. A motion state assessment device, characterized in that: include: An acquisition module is used to load video information, parse the image information of the current frame, and obtain the object and current feature points in the current frame based on the image information; A processing module, used for filtering out target current feature points that exist in both the current frame and the previous frame from the current feature points, wherein the target current feature points belong to the current feature points; A first evaluation module, configured to determine the motion state of the object based on a change in a direction angle of a vector between two current feature points of the target in the current frame and a direction angle of a corresponding vector in the previous frame; or, The motion state of the object is determined based on the displacement between the target current feature point in the current frame and the corresponding historical feature point in the previous frame, wherein the historical feature point is a feature point included in the previous frame.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method described in any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images