A method, apparatus, and electronic device for detecting objects with abnormal motion states.
By performing image segmentation and optical flow calculation on logistics transportation video frames, and combining clustering algorithms to detect abnormal motion states, the problem of inaccurate package loss detection in logistics transportation has been solved, achieving efficient and accurate detection results.
Patent Information
- Application Number
- CN202110804571.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-16
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-07-16
AI Technical Summary
Existing technologies are not accurate enough in detecting lost packages during logistics and transportation, and are affected by equipment and environment, leading to inaccurate detection.
By segmenting the video frames to be processed, determining the object segmentation map and reference region, calculating the target optical flow in combination with optical flow information, and using clustering algorithms to judge motion state anomalies, the accuracy and efficiency of detection are improved.
It enables real-time detection of objects with abnormal motion states, improving the accuracy and efficiency of detection, and is suitable for various application scenarios while reducing the demand for computing resources.
Smart Images

Figure CN113469993B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a method, device and electronic device for detecting an object with abnormal motion state. Background Art
[0002] With the vigorous development of the logistics industry, the transportation of massive parcels has become a new challenge. How to accurately detect the loss of parcels during transportation in real time has become a difficult problem that needs to be solved urgently in the industry.
[0003] At present, in order to solve the problem that manual labor cannot detect dropped goods in time, it is proposed to integrate a new control system and terminal detection method of dropped goods on existing equipment, as well as a method of detecting suspected dropped goods by relying on the movement laws of objects in fixed scenes.
[0004] However, the detection accuracy of the above method is affected by the equipment and environment. In the scenario where the object transportation equipment is transporting logistics objects, there is a problem of inaccurate detection of dropped goods. Summary of the invention
[0005] The present application provides a method, device and electronic device for detecting an object in an abnormal motion state, which are used to detect a target object in an abnormal motion state.
[0006] In a first aspect, the present application provides a method for detecting an object with abnormal motion state, the method comprising:
[0007] Perform image segmentation on each video frame in the video to be processed to obtain an object segmentation map corresponding to each video frame, wherein the object segmentation map includes each candidate object in the corresponding video frame;
[0008] Detecting the object transport equipment in each of the video frames to determine a reference area corresponding to each of the video frames; the reference area is determined based on the object transport equipment detected in the corresponding video frame;
[0009] Based on the optical flow information corresponding to each video frame, the object segmentation map and the reference area, determining the target optical flow information corresponding to each video frame, the target optical flow information is the optical flow information of the target object in the corresponding video frame, and the target object includes the candidate object carried by the object transport device in the corresponding video frame;
[0010] Based on the target optical flow information corresponding to each video frame, it is determined whether there is a target object with abnormal motion state in the video to be processed.
[0011] Through the above method, real-time detection of objects with abnormal motion states can be achieved, and objects with abnormal motion states can be automatically and conveniently detected in various application scenarios.
[0012] In a possible design, performing image segmentation on each video frame in the video data to be processed to obtain an object segmentation map corresponding to each video frame includes:
[0013] Acquire video data having a number greater than or equal to a preset number of frames as video data to be processed;
[0014] Perform image segmentation on each video frame in the video data to be processed to obtain an object segmentation map corresponding to each video frame.
[0015] Through the above method, the accuracy of detecting objects with abnormal motion states can be effectively improved, and the computing resources required in the process of detecting objects with abnormal motion states can be effectively saved.
[0016] In a possible design, determining the target optical flow information corresponding to each video frame based on the optical flow information corresponding to each video frame, the object segmentation map, and the reference area includes:
[0017] Calculating optical flow information of each video frame;
[0018] The optical flow information, the object segmentation map and the reference area corresponding to each of the video frames are superimposed to determine the target optical flow information corresponding to each of the video frames.
[0019] Determining the target optical flow information through the above method can accurately identify objects that may have abnormal motion states, and performing corresponding detection of objects with abnormal motion states can effectively improve detection efficiency and save computing resources.
[0020] In a possible design, determining the target optical flow information corresponding to each video frame based on the optical flow information corresponding to each video frame, the object segmentation map, and the reference area includes:
[0021] Superimposing the object segmentation map and the reference area corresponding to each video frame to determine the target object corresponding to each video frame;
[0022] The optical flow information of the target object is calculated to determine the target optical flow information corresponding to each video frame.
[0023] The above method can realize real-time detection of objects with abnormal motion status, effectively improve detection efficiency and save computing resources.
[0024] In a possible design, determining whether there is a target object with abnormal motion state in the video to be processed based on the target optical flow information corresponding to each video frame includes:
[0025] Obtaining accumulated optical flow information by accumulating target optical flow information corresponding to each video frame;
[0026] Clustering the accumulated optical flow information to obtain a clustering result;
[0027] Determine whether the number of cluster centers in the clustering result is greater than a preset threshold;
[0028] If yes, it is determined that there is a target object with abnormal motion state in the video to be processed, wherein the abnormal motion state indicates that the target object is separated from the object transport device;
[0029] If not, it is determined that there is no target object with abnormal motion state in the video to be processed.
[0030] The above method can be directly used to realize real-time detection of objects with abnormal motion states, and is applicable to different scenarios, which helps to improve the efficiency of detecting objects with abnormal motion states.
[0031] In a second aspect, the present application provides a device for detecting an object with an abnormal motion state, the device comprising:
[0032] A segmentation module performs image segmentation on each video frame in the video data to be processed to obtain an object segmentation map corresponding to each video frame; wherein the object segmentation map includes each candidate object in the corresponding video frame;
[0033] A positioning module detects the object transport equipment in each video frame and determines a reference area corresponding to each video frame; wherein the reference area is determined based on the object transport equipment detected in the corresponding video frame;
[0034] A calculation module, based on the optical flow information corresponding to each video frame, the object segmentation map and the reference area, determines the target optical flow information corresponding to each video frame, wherein the target optical flow information is the optical flow information of the target object in the corresponding video frame, and the target object includes the candidate object carried by the object transport device in the corresponding video frame;
[0035] The detection module determines whether there is a target object with abnormal motion state in the video to be processed based on the target optical flow information corresponding to each video frame.
[0036] In a possible design, the segmentation module is specifically used to obtain video data with a number greater than or equal to a preset number of frames as video data to be processed; perform image segmentation on each video frame in the video data to be processed to obtain an object segmentation map corresponding to each video frame.
[0037] In one possible design, the calculation module is specifically used to calculate the optical flow information of each video frame; based on superimposing the optical flow information corresponding to each video frame, the object segmentation map and the reference area, the target optical flow information corresponding to each video frame is determined.
[0038] In a possible design, the calculation module is specifically used to superimpose the object segmentation map and the reference area corresponding to each video frame to determine the target object corresponding to each video frame; calculate the optical flow information of the target object to determine the target optical flow information corresponding to each video frame.
[0039] In one possible design, the detection module is specifically used to accumulate the target optical flow information corresponding to each video frame to obtain cumulative optical flow information; cluster the accumulated optical flow information to obtain a clustering result; determine whether the number of cluster centers in the clustering result is greater than a preset threshold; if so, determine that there is a target object with an abnormal motion state in the video to be processed, wherein the abnormal motion state indicates that the target object is separated from the object transportation equipment; if not, determine that there is no target object with an abnormal motion state in the video to be processed.
[0040] In a third aspect, the present application provides an electronic device, the electronic device comprising:
[0041] Memory, used to store computer programs;
[0042] The processor is used to implement the above-mentioned method steps for detecting an object with abnormal motion state when executing the computer program stored in the memory.
[0043] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method steps for detecting an object with an abnormal motion state are implemented.
[0044] For each aspect from the second to the fourth aspect and the technical effects that may be achieved by each aspect, please refer to the above description of the technical effects that can be achieved by the first aspect or various possible schemes in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 A flowchart of a method for detecting an object with abnormal motion state provided by the present application;
[0046] Figure 2 A schematic diagram of a collected single-frame image provided in the present application;
[0047] Figure 3A schematic diagram of a generated object segmentation map provided in the present application;
[0048] Figure 4 A schematic diagram of a reference area map generated provided by the present application;
[0049] Figure 5 A schematic diagram of optical flow information of a whole image generated by the present application;
[0050] Figure 6 A schematic diagram of a first overlay image generated provided by the present application;
[0051] Figure 7 A schematic diagram of target optical flow information generated by the present application;
[0052] Figure 8 A schematic diagram of a second overlay image generated provided by the present application;
[0053] Fig. 9 A schematic diagram of accumulated optical flow information of a reference area provided in this application;
[0054] Fig.10 A schematic diagram of a clustering result provided for this application;
[0055] Fig.11 A schematic diagram of a device for detecting an object with abnormal motion state provided by the present application;
[0056] Fig.12 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION
[0057] The embodiments of the present application provide a method, device and electronic device for detecting objects with abnormal motion states, which realizes real-time detection of objects with abnormal motion states by clustering results based on target optical flow information corresponding to each video frame, and solves the problem that the existing technology applied to object transportation equipment cannot accurately detect objects with abnormal motion states during the transportation of logistics objects.
[0058] According to the method provided in the embodiment of the present application, firstly, a camera is used to obtain data to be processed that is greater than or equal to a preset number of frames, so as to realize real-time detection of objects with abnormal motion status. Compared with the existing method that relies on device integration, the deployment scenarios can be more diverse and the detection cost can be reduced. Then, image segmentation is performed on each video frame in the video to be processed to obtain an object segmentation map corresponding to each video frame, and the object segmentation map includes each candidate object in the corresponding video frame;
[0059] The object transportation equipment in each video frame is detected, and the reference area corresponding to each video frame is determined. Based on the optical flow information, object segmentation map and reference area corresponding to each video frame, the target optical flow information corresponding to each video frame is determined. Based on the target optical flow information corresponding to each video frame, it is determined whether there is a target object with an abnormal motion state in the video to be processed. Compared with the existing method that relies on detecting the motion laws of objects in fixed scenes, the accuracy of detecting objects with abnormal motion states is improved.
[0060] The method provided in the embodiments of the present application is further described in detail below in conjunction with the accompanying drawings.
[0061] See also Figure 1 As shown, the embodiment of the present application provides a method for detecting an object with abnormal motion state, and the specific process is as follows:
[0062] Step 11: Obtaining data to be processed that is greater than or equal to a preset number of frames;
[0063] Collect N consecutive frames of images of the same size as video data, where N represents a positive integer greater than or equal to 2, where each frame of the image collected can contain candidate objects and object transportation equipment. It should be noted that the candidate objects are of the same category, and the object area corresponding to the candidate objects is a whole area, and there are no objects of other categories in the object area. For example, the object can be represented as a package. Figure 2 As shown, the white part corresponds to the candidate object, the black part corresponds to the object transportation equipment, and the oblique line part corresponds to the image background.
[0064] Here, video data can be collected by an ordinary camera. Generally, the camera can obtain 25 frames of images per second. The abnormal state of the candidate object on the object transport device can last for 2 seconds, such as the abnormal state of falling goods. The embodiment of the present application takes the above general situation as an example, and does not elaborate on other situations.
[0065] In video data, the changes between each two consecutive frames are very small. Considering that if the number of frames of video data obtained is very small, the changes between N consecutive frames of images will also be small, that is to say, the information required to detect objects with abnormal motion states will be limited, and there is a problem of inaccurate results in detecting objects with abnormal motion states.
[0066] To solve the above problem, in the embodiment of the present application, a preset frame number is set to obtain a video to be processed with a frame number greater than or equal to the preset frame number, so as to improve the accuracy of detecting objects with abnormal motion states. The preset frame number can be adjusted accordingly according to the frame number of the acquired video data, and the preset frame number is set to 10 here.
[0067] Specifically, first acquire video data, and compare whether the frame number of the video data is greater than or equal to the preset frame number based on the preset frame number: if so, terminate the acquisition operation; if not, continue to acquire new video data until the frame number of all acquired video data is greater than or equal to the preset frame number, and terminate the acquisition operation.
[0068] Through the above method, not only the accuracy of detecting objects with abnormal motion states is effectively improved, but also the computing resources required in the process of detecting objects with abnormal motion states are effectively saved.
[0069] Step 12: Perform image segmentation on each video frame in the video to be processed to obtain an object segmentation map corresponding to each video frame;
[0070] Specifically, in the first model, each video frame is obtained, all object regions in each video frame are identified, and the corresponding object regions are segmented to generate an object segmentation map corresponding to each video frame. The object segmentation map generated corresponding to a single frame image is shown in FIG. Figure 3 As shown, the white area corresponds to the object information, and the oblique line area corresponds to the image background.
[0071] In addition, in the embodiment of the present application, the first model can be obtained by training a semantic segmentation network.
[0072] Step 13: Detecting the object transport equipment in each video frame to determine a reference area corresponding to each video frame;
[0073] Specifically, in the second model, each video frame is obtained, all reference areas in each video frame are detected, and the corresponding reference areas are marked with rectangular frames, the object transportation equipment in each video frame is determined, and the reference area of each video frame is determined based on the detected object transportation equipment, and a reference area map corresponding to each video frame is generated. The reference area map corresponding to a single frame image is shown in FIG. Figure 4 As shown, the area delineated by the black rectangular frame corresponds to the reference area.
[0074] In addition, in the embodiment of the present application, the second model can be obtained by deep learning detection network training.
[0075] Step 14: determining target optical flow information corresponding to each video frame based on the optical flow information corresponding to each video frame, the object segmentation map and the reference area;
[0076] In order to make the detection range of objects with abnormal motion states more targeted, save the calculation time of detecting objects with abnormal motion states, and improve the accuracy of detecting objects with abnormal motion states.
[0077] In an embodiment of the present application, the target optical flow information corresponding to each video frame is determined based on the calculation of the optical flow information corresponding to each video frame, as well as the object segmentation map and reference area obtained as described above, where the target optical flow information corresponds to the optical flow information of the target object in the video frame, and the target object corresponds to the candidate object carried by the object transport equipment.
[0078] For example, in the embodiment of the present application, the target optical flow information can be determined by two methods, and the two methods will be specifically described below:
[0079] The first method is to calculate the optical flow information of each video frame, and then superimpose the optical flow information, object segmentation map and reference area corresponding to each video frame to determine the target optical flow information corresponding to each video frame.
[0080] Specifically, in the third model, each video frame is obtained, the optical flow information of all feature points in each video frame is calculated, and the optical flow information in each video frame is extracted. The optical flow information of the whole image generated by calculating the single frame image is shown in Figure 5 As shown, the dotted distribution area corresponds to the optical flow information. In addition, the third model in the embodiment of the present application can be obtained through neural network training.
[0081] According to the optical flow information corresponding to each video frame, the optical flow information, the object segmentation map and the reference area map corresponding to each video frame are superimposed to generate a first superimposed map corresponding to each video frame. The first superimposed map corresponding to a single frame image is shown in FIG. Figure 6 shown.
[0082] Determine the target optical flow information of the first superimposed image corresponding to each video frame. For the target optical flow information corresponding to a single frame image, refer to Figure 7 As shown, the black rectangular box corresponds to the reference area, and the dotted area is the optical flow information in the object area.
[0083] The second method is to superimpose the object segmentation map and the reference area corresponding to each video frame to determine the target object corresponding to each video frame, and calculate the optical flow information of the target object according to the determined target object to determine the target optical flow information corresponding to each video frame.
[0084] Specifically, the object segmentation map and the reference region map corresponding to each video frame are superimposed to generate a second superimposed map. The second superimposed map corresponding to a single frame image is shown in FIG. Figure 8 As shown, the black rectangular box corresponds to the reference area, and the white area is the object area of the target object.
[0085] According to the second overlay image corresponding to each video frame, the target object in each video frame is determined, and the target optical flow information of the object area corresponding to the target object in each video frame is calculated. The target optical flow information corresponding to a single frame image can be found in Figure 7 As shown, the black rectangular box corresponds to the reference area, and the dotted area is the optical flow information in the object area.
[0086] In addition, in the embodiment of the present application, the target optical flow information of the target object in each video frame can be calculated by the KL optical flow method (Lucas–Kanade, an optical flow estimation algorithm of two-frame difference), and the specific calculation method is as follows:
[0087] The second overlay image corresponding to each frame of the video frame is used as the first image to be processed, and then each second overlay image generates n+1 images of different resolutions according to the Gaussian pyramid structure, wherein the nth layer corresponds to the first image, and the 0th layer corresponds to the n+1th image after the second overlay image is processed at a low resolution. When the Gaussian pyramid moves from the nth layer upward to the 0th layer, the image resolution corresponding to each layer decreases in sequence according to a preset ratio. For example, if the preset ratio is 2, the resolution of the n+1th image corresponding to the 0th layer is 400×400, and the resolution of the first image corresponding to the 1st layer is 200×200.
[0088] In order to save the calculation time of the KL optical flow method, the optical flow information of the target object in the n+1th image is first calculated, and then the optical flow information of the target object in the nth image, the n-1th image, and the 1st image is calculated in sequence, wherein the optical flow information of the target object is composed of the optical flow information of the feature points corresponding to the target object, and the optical flow information includes the optical flow velocity and the optical flow angle of the optical flow velocity.
[0089] The optical flow speed can be divided into horizontal and vertical directions. The optical flow speed of the feature point (k, l) in the n+1th image is calculated, where the horizontal optical flow speed is u i (k,l), the vertical optical flow velocity is v i (k, l). According to the optical flow velocity of the feature point in the n+1th image, the optical flow velocity of the feature point in the nth image, the n-1th image, ... the 1st image is calculated in sequence. In the embodiment of the present application, the optical flow velocity of the feature point in the n-i+2th image in the n-i+2th image is calculated according to the optical flow velocity of the feature point in the n-i+1th image as an example to illustrate the process of calculating the optical flow velocity in each layer in sequence. The specific calculation method is as follows:
[0090] By using the bilinear interpolation method, the compensation optical flow of the feature points in the n-i+1th image is estimated according to the optical flow speed of the feature points in the n-i+1th image, and the compensation optical flow is geometrically expanded according to the preset ratio to obtain the updated compensation optical flow. The compensation optical flow of the feature points in the n-i+2th image is divided into horizontal and vertical directions, which are respectively expressed as and Then, based on the compensated optical flow, the compensated optical flow of the feature points in the n-i+2th image is obtained. The compensated optical flow is divided into horizontal and vertical directions, which are represented by u' i-1 (k,l) and v' i-1 (k,l).
[0091] According to the compensated optical flow value and the compensated optical flow value of the feature point in the n-i+2th image, the optical flow speed in the horizontal and vertical directions of the feature point in the n-i+2th image is calculated. The specific calculation method is shown in the following formula 1:
[0092]
[0093] Among them, u i-1 (k,l) represents the horizontal optical flow velocity of the feature point in the n-i+2th image, Indicates the horizontal compensation optical flow of the feature point in the n-i+2th image, u' i-1 (k,l) represents the compensated optical flow of the feature point in the horizontal direction of the n-i+2th image, v i-1 (k,l) represents the optical flow velocity in the vertical direction of the feature point in the n-i+2th image, Indicates the vertical compensation optical flow of the feature point in the n-i+2th image, v' i-1 (k,l) represents the compensated optical flow in the horizontal direction of the feature point in the n-i+2th image.
[0094] The optical flow velocities of the corresponding feature points in the n+1 images corresponding to the second overlay image are summed and averaged to obtain the optical flow velocity of the feature points in the second overlay image, so as to determine the optical flow velocity of the target object in each video frame. The specific calculation method is shown in the following formula 2:
[0095]
[0096] in, represents the horizontal optical flow velocity of the feature point (k, l) of the second superimposed image, Represents the vertical optical flow velocity of the feature point (k, l) of the second superimposed image.
[0097] Based on the above optical flow velocities in the horizontal and vertical directions, the optical flow velocity value I of the feature point in the second superposition image is calculated. i, the specific calculation method is shown in the following formula 3:
[0098]
[0099] Where i represents the i-th frame image in the video to be processed, I i Represents the optical flow velocity of the feature point in the second overlay image.
[0100] The optical flow angle can be divided into two directions: horizontal and vertical. For example, the specific calculation method for calculating the optical flow angle of the feature point in the n-i+1th image is shown in the following formula 4:
[0101]
[0102] Among them, i represents the i-th frame image in the video to be processed, θ i represents the optical flow angle of the feature point in the second superposition image, v yi Indicates the vertical optical flow velocity of the feature point in the second superposition image, v xi Indicates the horizontal optical flow velocity of the feature point in the second overlay image.
[0103] The KL optical flow method is adopted to obtain the target optical flow information of the second overlay image based on the optical flow speed and optical flow angle of the second overlay image, and the target optical flow information of the target object in each video frame is determined. Only the target optical flow information of the feature points in the image is calculated, which not only saves computing costs but also speeds up the processing speed.
[0104] Step 15: Based on the target optical flow information corresponding to each video frame, determine whether there is a target object with an abnormal motion state in the video to be processed.
[0105] In order to improve the efficiency and accuracy of detecting objects with abnormal motion states, firstly, the target optical flow information corresponding to each video frame of a preset number of frames is accumulated to obtain the accumulated optical flow information, then the accumulated optical flow information is clustered to obtain the clustering result, and finally, it is determined whether the number of cluster centers in the clustering result is greater than a preset threshold: if so, it is determined that there is a target object with abnormal motion state in the video to be processed; if not, it is determined that there is no target object with abnormal motion state in the video to be processed. In addition, the abnormal motion state indicates that the target object is separated from the object transportation device.
[0106] For example, first, select a preset number of consecutive video frames in the video to be processed, accumulate the target optical flow information corresponding to the video frames, and obtain the accumulated optical flow information of the feature points in each video frame, which is used to represent the motion trajectory of a target object that may have an abnormal motion state within the time corresponding to each of the preset number of video frames. In addition, the accumulated optical flow information is divided into accumulated optical flow angle and accumulated optical flow speed. The accumulated optical flow information of a reference area in a single frame image can be found in Fig. 9 As shown, the scattered line segments correspond to the accumulated optical flow information, and the black rectangular box corresponds to the specific area.
[0107] Specifically, the specific method for calculating the optical flow angle of the accumulated optical flow information is shown in the following formula 5:
[0108]
[0109] Where M represents the preset number of frames, θ represents the optical flow angle of the target optical flow information corresponding to the feature point in the video frame, and θ i Represents the optical flow angle of the target optical flow information corresponding to the feature point in the i-th frame.
[0110] The specific method for calculating the optical flow speed of the accumulated optical flow information is shown in the following formula 6:
[0111]
[0112] Where M represents the preset frame number, I represents the optical flow speed of the target optical flow information corresponding to the feature point in the video frame, and I i Represents the optical flow speed of the target optical flow information corresponding to the feature point in the i-th frame.
[0113] Then, the accumulated optical flow information is clustered. In order to improve the accuracy of clustering, a distance parameter is introduced in the embodiment of the present application to compare with the optical flow angle of the accumulated optical flow information to determine whether the motion trajectory of each feature point is consistent, so as to detect whether there is a target object with abnormal motion state in each video frame. A cluster point number threshold can also be introduced here to identify noise points through the cluster point number threshold to eliminate the influence of noise points on the clustering results.
[0114] For example, the accumulated optical flow information of the feature points is clustered by the density clustering algorithm of DBSCAN (Density-Based Spatial Clustering of Applications with Noise) to obtain a clustering result, and it is determined whether the number of cluster centers in the clustering result is greater than a preset threshold: if so, it is determined that there is a target object with an abnormal motion state in the video to be processed; if not, it is determined that there is no target object with an abnormal motion state in the video to be processed.
[0115] First, obtain the distance parameter and the cluster point number threshold. The distance parameter represents the radius of the neighborhood of a given object in the cluster. The given object represents a feature point that has not been clustered in the cluster. The cluster point number threshold is used to characterize the minimum number of points in the neighborhood that need to be reached for the core object. The distance parameter and the cluster point number threshold can be set arbitrarily. For example, the distance parameter is set to 15 degrees (15°) and the cluster point number threshold is set to: the average area of the reference area in each video frame of the preset number of frames to which the target object belongs.
[0116] Then, the DBSCAN density clustering algorithm is used to cluster the accumulated optical flow information of the corresponding feature points of the target object in each video frame of the preset frame number. Here, the optical flow speed of the accumulated optical flow information is used as the abscissa of the actual clustering space, and the optical flow angle of the accumulated optical flow information is used as the ordinate of the actual clustering space.
[0117] In the actual clustering space, the feature point without clustering is set as the given object as the center, and the distance parameter is set as the radius to construct the neighborhood of the given object. If the number of feature points falling in the neighborhood is greater than the threshold of the number of cluster points, it is considered that the feature points in the neighborhood belong to the category of the given object; if the number of feature points falling in the neighborhood is not greater than the threshold of the number of cluster points, it is considered that the feature points in the neighborhood do not belong to the category of the given object.
[0118] Finally, through the clustering operation, the corresponding feature points of the target object in each video frame of the preset number of frames can be classified into categories to obtain the number of divided categories, that is, the number of cluster centers. Here, the actual clustering space information is mapped to the position coordinate space, and the clustering results are shown in Fig.10 As shown, the black circular area corresponds to the cluster center, and the scattered line segments correspond to the optical flow information.
[0119] Based on the above clustering results, it is determined whether the number of cluster centers in the clustering results is greater than a preset threshold: if so, it is determined that there is a target object with an abnormal motion state in the video to be processed; if not, it is determined that there is no target object with an abnormal motion state in the video to be processed. In addition, the abnormal motion state indicates that the target object is separated from the object transportation equipment, for example, the target object falls off during transportation.
[0120] The above method can be used to directly detect target objects with abnormal motion states, which helps to improve the efficiency of detecting objects with abnormal motion states.
[0121] Based on the same inventive concept, the present application also provides a device for detecting an object with abnormal motion state, which is used to detect a target object with abnormal motion state, solves the problem of inaccurate detection of objects with abnormal motion state in the current scene of object transportation equipment transporting logistics objects, and effectively improves the accuracy of detecting objects with abnormal motion state, see Fig.11 , the device comprises:
[0122] The segmentation module 111 performs image segmentation on each video frame in the video data to be processed to obtain an object segmentation map corresponding to each video frame; wherein the object segmentation map includes each candidate object in the corresponding video frame;
[0123] The positioning module 112 detects the object transport equipment in each video frame and determines a reference area corresponding to each video frame; wherein the reference area is determined based on the object transport equipment detected in the corresponding video frame;
[0124] The calculation module 113 determines the target optical flow information corresponding to each video frame based on the optical flow information corresponding to each video frame, the object segmentation map and the reference area, wherein the target optical flow information is the optical flow information of the target object in the corresponding video frame, and the target object includes the candidate object carried by the object transport device in the corresponding video frame;
[0125] The detection module 114 determines whether there is a target object with abnormal motion state in the video to be processed based on the target optical flow information corresponding to each video frame.
[0126] In a possible design, the segmentation module 111 is specifically used to obtain video data with a number greater than or equal to a preset number of frames as video data to be processed; perform image segmentation on each video frame in the video data to be processed to obtain an object segmentation map corresponding to each video frame.
[0127] In a possible design, the calculation module 113 is specifically used to calculate the optical flow information of each video frame; based on superimposing the optical flow information, object segmentation map and reference area corresponding to each video frame, determine the target optical flow information corresponding to each video frame.
[0128] In a possible design, the calculation module 113 is specifically used to superimpose the object segmentation map and the reference area corresponding to each video frame to determine the target object corresponding to each video frame; calculate the optical flow information of the target object to determine the target optical flow information corresponding to each video frame.
[0129] In one possible design, the detection module 114 is specifically used to accumulate the target optical flow information corresponding to each video frame to obtain cumulative optical flow information; cluster the accumulated optical flow information to obtain a clustering result; determine whether the number of cluster centers in the clustering result is greater than a preset threshold; if so, determine that there is a target object with an abnormal motion state in the video to be processed, wherein the abnormal motion state indicates that the target object is separated from the object transportation equipment; if not, determine that there is no target object with an abnormal motion state in the video to be processed.
[0130] Based on the above device, target objects in abnormal motion states are detected, which solves the current problem of low accuracy in detecting objects in abnormal motion states, and realizes automatic and convenient detection of objects in abnormal motion states in a variety of real application scenarios.
[0131] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the electronic device can realize the function of the aforementioned device for detecting an object with abnormal motion state, referring to Fig.12 , the electronic device comprises:
[0132] At least one processor 121, and a memory 122 connected to the at least one processor 121. The specific connection medium between the processor 121 and the memory 122 is not limited in the embodiment of the present application. Fig.12 In the example, the processor 121 and the memory 122 are connected via the bus 120. The bus 120 is Fig.12 The connection between other components is shown by bold lines, and is not intended to be limiting. The bus 120 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 121 can also be called a controller, and there is no limitation on the name.
[0133] In the embodiment of the present application, the memory 122 stores instructions that can be executed by at least one processor 121. The at least one processor 121 can execute the method for detecting an object with abnormal motion state discussed above by executing the instructions stored in the memory 122. The processor 121 can implement Fig.12 The functions of each module in the device shown.
[0134] Among them, the processor 121 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 122 and calling data stored in the memory 122, the various functions of the device and process data, the device can be monitored as a whole.
[0135] In one possible design, the processor 121 may include one or more processing units, and the processor 121 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 121. In some embodiments, the processor 121 and the memory 122 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.
[0136] The processor 121 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the object method for detecting abnormal motion state disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.
[0137] The memory 122 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 122 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 122 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 122 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0138] By programming the processor 121, the code corresponding to the object method for detecting abnormal motion state described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 1The steps of the method for detecting an object with abnormal motion state in the embodiment shown are as follows: How to design and program the processor 121 is a technique known to those skilled in the art and will not be described in detail here.
[0139] Based on the same inventive concept, an embodiment of the present application further provides a storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the method for detecting an object with abnormal motion state discussed above.
[0140] In some possible implementations, various aspects of the method for detecting an object with an abnormal motion state provided by the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of the method for detecting an object with an abnormal motion state according to various exemplary implementations of the present application described above in this specification.
[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0142] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0145] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A method for detecting an object with abnormal motion state, It is characterized in that The method comprises: Perform image segmentation on each video frame in the video to be processed to obtain an object segmentation map corresponding to each video frame, wherein the object segmentation map includes each candidate object in the corresponding video frame; Detecting the object transport equipment in each of the video frames to determine a reference area corresponding to each of the video frames; the reference area is determined based on the object transport equipment detected in the corresponding video frame; Based on the optical flow information corresponding to each video frame, the object segmentation map and the reference area, determining the target optical flow information corresponding to each video frame, the target optical flow information is the optical flow information of the target object in the corresponding video frame, and the target object includes the candidate object carried by the object transport device in the corresponding video frame; Based on the target optical flow information corresponding to each video frame, it is determined whether there is a target object with abnormal motion state in the video to be processed.
2. The method according to claim 1, It is characterized in that The step of performing image segmentation on each video frame in the video data to be processed to obtain an object segmentation map corresponding to each video frame includes: Acquire video data having a number greater than or equal to a preset number of frames as video data to be processed; Perform image segmentation on each video frame in the video data to be processed to obtain an object segmentation map corresponding to each video frame.
3. The method according to claim 1, It is characterized in that The determining, based on the optical flow information corresponding to each video frame, the object segmentation map and the reference area, the target optical flow information corresponding to each video frame comprises: Calculating optical flow information of each video frame; The optical flow information, the object segmentation map and the reference area corresponding to each of the video frames are superimposed to determine the target optical flow information corresponding to each of the video frames.
4. The method according to claim 1, It is characterized in that The determining, based on the optical flow information corresponding to each video frame, the object segmentation map and the reference area, the target optical flow information corresponding to each video frame comprises: Superimposing the object segmentation map and the reference area corresponding to each video frame to determine the target object corresponding to each video frame; The optical flow information of the target object is calculated to determine the target optical flow information corresponding to each video frame.
5. The method according to any one of claims 1 to 4, It is characterized in that The determining whether there is a target object with abnormal motion state in the video to be processed based on the target optical flow information corresponding to each video frame includes: Obtaining accumulated optical flow information by accumulating target optical flow information corresponding to each video frame; Clustering the accumulated optical flow information to obtain a clustering result; Determine whether the number of cluster centers in the clustering result is greater than a preset threshold; If yes, it is determined that there is a target object with abnormal motion state in the video to be processed, wherein the abnormal motion state indicates that the target object is separated from the object transport device; If not, it is determined that there is no target object with abnormal motion state in the video to be processed.
6. A device for detecting an object with abnormal motion state, It is characterized in that The device comprises: A segmentation module performs image segmentation on each video frame in the video data to be processed to obtain an object segmentation map corresponding to each video frame; wherein the object segmentation map includes each candidate object in the corresponding video frame; A positioning module detects the object transport equipment in each video frame and determines a reference area corresponding to each video frame; wherein the reference area is determined based on the object transport equipment detected in the corresponding video frame; A calculation module, based on the optical flow information corresponding to each video frame, the object segmentation map and the reference area, determines the target optical flow information corresponding to each video frame, wherein the target optical flow information is the optical flow information of the target object in the corresponding video frame, and the target object includes the candidate object carried by the object transport device in the corresponding video frame; The detection module determines whether there is a target object with abnormal motion state in the video to be processed based on the target optical flow information corresponding to each video frame.
7. The device according to claim 6, It is characterized in that The calculation module is specifically used to calculate the optical flow information of each video frame; based on superimposing the optical flow information corresponding to each video frame, the object segmentation map and the reference area, determine the target optical flow information corresponding to each video frame.
8. The device according to claim 6, It is characterized in that The detection module is specifically used to accumulate the target optical flow information corresponding to each video frame to obtain the accumulated optical flow information; cluster the accumulated optical flow information to obtain a clustering result; and determine whether the number of cluster centers in the clustering result is greater than a preset threshold; If so, it is determined that there is a target object with an abnormal motion state in the video to be processed, wherein the abnormal motion state indicates that the target object is separated from the object transportation device; if not, it is determined that there is no target object with an abnormal motion state in the video to be processed.
9. An electronic device, It is characterized in that include: Memory, used to store computer programs; A processor, configured to implement the method steps of any one of claims 1 to 5 when executing the computer program stored in the memory.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 5 are implemented.
