Violent sorting detection method and device, terminal and computer readable storage medium

Abstract

The application provides a violent sorting detection method, device, terminal and computer readable storage medium, the violent sorting detection method comprises: determining a parallax map and an optical flow map of a current video frame based on the current video frame and historical video frames before the current video frame; performing semantic segmentation on the historical video frames before the current video frame to determine a region image of a target object; determining position variable information of the target object based on the region image of the target object, the parallax map and the optical flow map corresponding to the current video frame; and determining whether the behavior of the sorting target object in the current video frame is violent sorting based on the position variable information of the target object. The application determines the real motion of the target object in the real three-dimensional scene through the position variable information of the target object, can solve the problem that it is difficult to obtain stable target object motion information due to the principle that the image collected by the camera is large near and small far, and further improves the detection accuracy of the behavior of the sorting target object.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, terminal, and computer-readable storage medium for detecting violent sorting. Background Technology

[0002] With the development of emerging industries such as e-commerce and online shopping, the demand for express delivery services is constantly increasing, leading to explosive growth in my country's express logistics industry. Express delivery has become an increasingly common logistics method between businesses and consumers. The express delivery industry is a vital public service connecting millions of businesses and the general public, playing an increasingly important role in promoting the transformation of distribution methods and facilitating consumption upgrades. In recent years, while the service quality and delivery efficiency of my country's express delivery industry have been continuously improving, violent sorting practices remain rampant. These violent sorting problems not only harm the interests of consumers and businesses but also damage the competitiveness of express delivery companies and the reputation of the industry. Summary of the Invention

[0003] The main technical problem solved by this invention is to provide a method, device, terminal and computer-readable storage medium for detecting violent sorting, thereby solving the problem of low detection accuracy of violent sorting behavior in the prior art.

[0004] To address the aforementioned technical problems, the first technical solution adopted by this invention is: providing a method for detecting violent sorting, the method comprising: determining the disparity map and optical flow map corresponding to the current video frame based on the acquired current video frame and historical video frames preceding the current video frame; both the current video frame and the historical video frames are multi-view images captured by each lens of a multi-view camera at different times, and the multi-view images contain target objects; performing semantic segmentation on the historical video frames preceding the current video frame to determine the region image of the target object; determining the positional variable information of the target object based on the region image of the target object, the disparity map and optical flow map corresponding to the current video frame; and determining whether violent sorting exists in the behavior of sorting the target object in the current video frame based on the positional variable information of the target object.

[0005] The multi-view image includes at least a left-view image and a right-view image; both the left-view image and the right-view image contain the target object; based on the acquired current video frame and the historical video frames preceding the current video frame, the disparity map and optical flow map corresponding to the current video frame are determined, including: acquiring the current video frame; in response to the current video frame being a non-first frame image, obtaining the disparity map of the current video frame based on the left-view image and the right-view image corresponding to the current video frame; generating the optical flow map corresponding to the current video frame based on the left-view image of the current video frame and the left-view images of the historical video frames adjacent to the current video frame; or; generating the optical flow map corresponding to the current video frame based on the right-view image of the current video frame and the right-view images of the historical video frames adjacent to the current video frame.

[0006] Specifically, the disparity map of the current video frame is obtained based on the left and right eye images corresponding to the current video frame, including: using a semi-global matching algorithm to obtain the disparity map of the current video frame based on the left and right eye images corresponding to the current video frame.

[0007] The process of semantic segmentation of historical video frames preceding the current video frame to determine the region image of the target object includes: using a semantic segmentation network to segment the left or right eye image in historical video frames adjacent to the current video frame to determine the region image of the target object.

[0008] The process of determining the target object's positional variable information based on the target object's region image, disparity map, and optical flow map corresponding to the current video frame includes: extracting the target object's optical flow information from the optical flow map corresponding to the current video frame and extracting the target object's disparity information from the disparity map corresponding to the current video frame based on the target object's region image; determining the target object's positional information at a first moment based on the target object's disparity information in historical video frames adjacent to the current video frame and the target object's region image; the first moment being the acquisition time of the historical video frame adjacent to the current video frame; determining the target object's positional information at a second moment based on the target object's optical flow information and the target object's disparity information in the current video frame; the second moment being later than the first moment; the second moment being the acquisition time of the current video frame; and determining the target object's positional variable information based on the target object's positional information at the first moment and the second moment.

[0009] The optical flow information of the target object includes the target object's motion distance. Based on the disparity information and region image of the target object in historical video frames adjacent to the current video frame, the position information of the target object at the first moment is determined. Prior to this, it also includes: determining whether the target object's motion distance exceeds a preset distance; if the target object's motion distance exceeds the preset distance, the target object is identified as a candidate target; if the target object's motion distance does not exceed the preset distance, the target object is eliminated. The position information of the target object at the first moment is determined based on the disparity information and region image of the target object in historical video frames adjacent to the current video frame, including: determining the position information of the candidate target at the first moment based on the disparity information, region image, and device parameters of the candidate targets in historical video frames adjacent to the current video frame; the position information of the target object at the second moment is determined based on the optical flow information and disparity information of the target object in the current video frame, including: determining the position information of the candidate target at the second moment based on the optical flow information, disparity information, and device parameters of the candidate targets; the position information is the coordinate position in the world coordinate system.

[0010] The location variable information includes the direction of movement and the relative distance of movement. Based on the location variable information of the target object, it is determined whether the sorting behavior of the target object in the current video frame is violent sorting, including: generating a relative distance histogram of the target object based on the relative distance of each pixel corresponding to the target object; and determining whether the sorting behavior of the target object in the current video frame is violent sorting based on the relative distance histogram of the target object.

[0011] The process of determining whether the sorting behavior of the target object in the current video frame involves violent sorting based on the relative distance histogram of the target object. This includes: using a violent sorting detection network to determine the probability value of violent sorting of the target object based on the relative distance histogram of the target object; wherein the violent sorting detection network is trained by a sample histogram obtained by weighting and summing the relative distance histograms corresponding to sample images of a consecutive preset number of frames, and the sample histogram is associated with labeled behavior categories, including violent sorting and non-violent sorting; if the probability value corresponding to the target object exceeds a preset value, it is determined that the sorting behavior of the target object in the current video frame involves violent sorting; if the probability value corresponding to the target object does not exceed the preset value, it is determined that the sorting behavior of the target object in the current video frame does not involve violent sorting.

[0012] To address the aforementioned technical problems, the second technical solution adopted by this invention is: providing a violent sorting detection device, comprising: an acquisition module, used to determine the disparity map and optical flow map corresponding to the current video frame based on the acquired current video frame and historical video frames preceding the current video frame; the current video frame and historical video frames are multi-view images captured by each lens of a multi-view camera at different times, and the multi-view images contain target objects; a segmentation module, used to perform semantic segmentation on the historical video frames preceding the current video frame to determine the region image of the target object; an analysis module, used to determine the positional variable information of the target object based on the region image of the target object, the disparity map and optical flow map corresponding to the current video frame; and a processing module, used to determine whether violent sorting occurs in the current video frame based on the positional variable information of the target object.

[0013] To solve the above-mentioned technical problems, the third technical solution adopted by the present invention is to provide a terminal, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. The processor is used to execute program data to implement the steps in the above-mentioned violent sorting detection method.

[0014] To solve the above-mentioned technical problems, the fourth technical solution adopted by the present invention is to provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps in the above-mentioned violent sorting detection method.

[0015] The beneficial effects of this invention are as follows: Unlike existing technologies, this invention provides a method, apparatus, terminal, and computer-readable storage medium for detecting violent sorting. The violent sorting detection method includes: determining the disparity map and optical flow map corresponding to the current video frame based on the acquired current video frame and historical video frames preceding the current video frame; both the current video frame and historical video frames are multi-view images captured by various lenses of a multi-view camera at different times, and the multi-view images contain target objects; performing semantic segmentation on the historical video frames preceding the current video frame to determine the region image of the target object; determining the position variable information of the target object based on the region image of the target object, the disparity map, and the optical flow map corresponding to the current video frame; and determining whether violent sorting occurs in the current video frame based on the position variable information of the target object. This application determines the position variable information of the target object based on the disparity map, optical flow map, and region image of the target object corresponding to the current video frame. By using the position variable information of the target object, the actual motion of the target object in a real 3D scene can be determined, which can greatly solve the problem of difficulty in obtaining stable target object motion information due to the principle of near-to-far image size, thereby improving the detection accuracy of the sorting behavior of the target object. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating an embodiment of the violent sorting detection method provided by the present invention;

[0018] Figure 2 This is a flowchart illustrating a specific embodiment of the violent sorting detection method provided by the present invention;

[0019] Figure 3 yes Figure 2 A flowchart illustrating a specific embodiment of step S201 in the provided violent sorting detection method;

[0020] Figure 4 yes Figure 2 A flowchart illustrating a specific embodiment of step S202 in the provided violent sorting detection method;

[0021] Figure 5 This is a schematic diagram of a specific embodiment of the violent sorting detection network provided by the present invention;

[0022] Figure 6 yes Figure 2A flowchart illustrating a specific embodiment of step S203 in the provided violent sorting detection method;

[0023] Figure 7 This is a schematic diagram of a specific embodiment of the violent sorting detection method provided by the present invention;

[0024] Figure 8 This is a schematic diagram of the framework of an embodiment of the violent sorting detection device provided by the present invention;

[0025] Figure 9 This is a schematic diagram of the framework of one embodiment of the terminal of this application;

[0026] Figure 10 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0027] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0028] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0029] In this article, the term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "more" in this article means two or more objects.

[0030] To enable those skilled in the art to better understand the technical solution of the present invention, the following describes in further detail a violent sorting detection method provided by the present invention in conjunction with the accompanying drawings and specific embodiments.

[0031] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the violent sorting detection method provided by the present invention.

[0032] This embodiment provides a method for detecting violent sorting, which includes the following steps.

[0033] S11: Based on the acquired current video frame and the historical video frames before the current video frame, determine the disparity map and optical flow map corresponding to the current video frame.

[0034] Specifically, the current video frame and the historical video frames are multi-view images captured by each lens of the multi-view camera at different times, and the images captured by each lens in the multi-view image all contain the target object. The multi-view image includes at least a left-view image and a right-view image, and both the left-view image and the right-view image contain the target object.

[0035] In one embodiment, the current video frame is acquired; in response to the current video frame being a non-first frame image, a disparity map of the current video frame is obtained based on the left and right eye images corresponding to the current video frame; an optical flow map corresponding to the current video frame is generated based on the left eye image of the current video frame and the left eye images of adjacent historical video frames. Alternatively, the optical flow map corresponding to the current video frame can be generated based on the right eye image of the current video frame and the right eye images of adjacent historical video frames.

[0036] In one embodiment, a semi-global matching algorithm is used to obtain the disparity map of the current video frame based on the left and right eye images corresponding to the current video frame.

[0037] S12: Perform semantic segmentation on historical video frames preceding the current video frame to determine the region image of the target object.

[0038] Specifically, a semantic segmentation network is used to segment the left or right eye images in historical video frames adjacent to the current video frame to determine the region image of the target object.

[0039] S13: Determine the positional variable information of the target object based on the region image of the target object, the disparity map and optical flow map corresponding to the current video frame.

[0040] Specifically, based on the region image of the target object, optical flow information of the target object is extracted from the optical flow map corresponding to the current video frame, and disparity information of the target object is extracted from the disparity map corresponding to the current video frame; based on the disparity information of the target object in the historical video frames adjacent to the current video frame and the region image of the target object, the position information of the target object at the first moment is determined; the first moment is the acquisition time of the historical video frame adjacent to the current video frame; based on the optical flow information of the target object and the disparity information of the target object in the current video frame, the position information of the target object at the second moment is determined; the second moment is later than the first moment; the second moment is the acquisition time of the current video frame; based on the position information of the target object at the first moment and the position information at the second moment, the position variable information of the target object is determined.

[0041] In one embodiment, the optical flow information of the target object includes the target object's motion distance. It is determined whether the target object's motion distance exceeds a preset distance; if the target object's motion distance exceeds the preset distance, the target object is identified as a candidate target; if the target object's motion distance does not exceed the preset distance, the target object is discarded. Based on the disparity information of candidate targets in historical video frames adjacent to the current video frame, the regional image of the candidate targets, and device parameters, the position information of the candidate targets at a first time point is determined; based on the optical flow information of the candidate targets, the disparity information of candidate targets in the current video frame, and device parameters, the position information of the candidate targets at a second time point is determined; the position information is a coordinate position in the world coordinate system.

[0042] S14: Based on the location variable information of the target object, determine whether the sorting behavior of the target object in the current video frame is violent sorting.

[0043] Specifically, the positional variable information includes the direction of movement and the relative distance of movement. Based on the relative distance of each pixel corresponding to the target object, a relative distance histogram of the target object is generated; based on the relative distance histogram of the target object, it is determined whether the sorting behavior of the target object in the current video frame involves violent sorting.

[0044] In one embodiment, a violent sorting detection network is used to determine the probability value of violent sorting of a target object based on the relative distance histogram of the target object. The violent sorting detection network is trained by a sample histogram obtained by weighting and summing the relative distance histograms corresponding to sample images of a consecutive preset number of frames. The sample histogram is associated with labeled behavior categories, including violent sorting and non-violent sorting. If the probability value corresponding to the target object exceeds a preset value, it is determined that violent sorting exists in the current video frame; if the probability value corresponding to the target object does not exceed the preset value, it is determined that violent sorting does not exist in the current video frame.

[0045] The violent sorting detection method provided in this embodiment includes: determining the disparity map and optical flow map corresponding to the current video frame based on the acquired current video frame and historical video frames before the current video frame; both the current video frame and the historical video frames are multi-view images captured by each lens of a multi-view camera at different times, and the multi-view images contain target objects; performing semantic segmentation on the historical video frames before the current video frame to determine the region image of the target object; determining the position variable information of the target object based on the region image of the target object, the disparity map and optical flow map corresponding to the current video frame; and determining whether violent sorting occurs in the current video frame based on the position variable information of the target object. This application determines the position variable information of the target object based on the disparity map, optical flow map and region image of the target object corresponding to the current video frame. By using the position variable information of the target object, the actual motion of the target object in a real 3D scene can be determined, which can greatly solve the problem of difficulty in obtaining stable target object motion information due to the principle of near-to-far image size in camera-acquired images, thereby improving the detection accuracy of the sorting behavior of target objects.

[0046] Please see Figure 2 , Figure 2 This is a flowchart illustrating a specific embodiment of the violent sorting detection method provided by the present invention. This embodiment provides a violent sorting detection method, which includes the following steps.

[0047] S201: Training the optical flow estimation network.

[0048] Specifically, the steps for training the optical flow estimation network are as follows.

[0049] Please see Figure 3 , Figure 3 yes Figure 2 A flowchart illustrating a specific embodiment of step S201 in the provided violent sorting detection method.

[0050] S2011: Obtain the first training sample set.

[0051] Specifically, the first training sample set includes multiple training data groups, each consisting of two consecutive frames of stereo image training samples. Each stereo image training sample includes a left-eye image and a right-eye image, both containing the same target. Each training data group is associated with a corresponding labeled optical flow map. In one embodiment, the first training sample set can also be an open-source optical flow dataset.

[0052] S2012: The optical flow estimation network is used to process each training data group to obtain the predicted optical flow map corresponding to each training data group.

[0053] In this embodiment, the left eye image in the binocular images is set as the reference. Two consecutive frames of binocular images from the training data set are input into the optical flow estimation network. The optical flow estimation network detects the positional changes of the target in the two consecutive binocular images in the two left eye images, and obtains the predicted optical flow map corresponding to the training data set.

[0054] In another embodiment, the optical flow estimation network can be set to use the right eye image in the binocular images as a reference, and detect the positional changes of the target in the two consecutive binocular images in the two right eye images to obtain the predicted optical flow map corresponding to the training data set.

[0055] S2013: The optical flow estimation network is trained iteratively based on the error between the predicted optical flow map and the labeled optical flow map corresponding to the same training data set.

[0056] Specifically, the error value between the predicted optical flow map and the labeled optical flow map corresponding to the same training data group is calculated by the optical flow estimation network, and the optical flow estimation network is iteratively trained based on the error value.

[0057] In one optional embodiment, the results of the optical flow estimation network are backpropagated, and the weights of the optical flow estimation network are corrected based on the error value between the predicted optical flow map and the labeled optical flow map of the same training data set, thereby training the optical flow estimation network.

[0058] The training data set is input into the optical flow estimation network, which detects changes in the position of the target object in the training data set within the left eye image. Training of the optical flow estimation network stops when the error between the predicted and labeled optical flow maps in the training data set is less than a preset threshold (which can be set manually, such as 1% or 5%).

[0059] S202: Training the semantic segmentation network.

[0060] Specifically, the semantic segmentation network provided in this embodiment can be a U-Net network. The U-Net network has a shorter processing time and can process the acquired images in real time.

[0061] Specifically, the steps for training the semantic segmentation network are as follows.

[0062] Please see Figure 4 , Figure 4 yes Figure 2 A flowchart illustrating a specific embodiment of step S202 in the provided violent sorting detection method.

[0063] S2021: Obtain the second training sample set.

[0064] Specifically, the second training sample set includes multiple sample images containing moving targets. These moving targets can be express parcels, pedestrians, etc. Each sample image is associated with a corresponding ground truth mask. In one embodiment, video data from an express parcel sorting scenario is collected, and multiple frames are selected from the video data as sample images. Each sample image contains express parcels, and the regions containing express parcels in each sample image are labeled to obtain the corresponding ground truth mask for each sample image. The ground truth mask is obtained by semantically segmenting each express parcel in the sample image.

[0065] S2022: Semantic segmentation of the sample image is performed using a semantic segmentation network to obtain the prediction mask image corresponding to the sample image.

[0066] In this embodiment, moving targets in the sample image are detected to obtain the position information of each moving target. Based on the position information of the moving targets and the relationship between the pixel value of the pixel at the corresponding coordinate position in the sample image and the pixel threshold, the value assigned to the pixel at that coordinate position is determined, thereby generating a prediction mask image corresponding to the sample image.

[0067] In another embodiment, the pixel value of each pixel in the sample image is directly compared with a pixel threshold. If a pixel value is greater than the pixel threshold, the corresponding pixel is assigned a value of 1. If a pixel value is less than the pixel threshold, the corresponding pixel is assigned a value of 0. Once the pixel values ​​of all pixels in the sample image have been compared with the pixel threshold, a prediction mask image corresponding to the sample image is obtained. The prediction mask image displays the position and contour information of the moving target.

[0068] S2023: The semantic segmentation network is trained iteratively based on the error between the predicted mask and the real mask corresponding to the same sample image.

[0069] Specifically, the error value between the predicted mask and the real mask corresponding to the same sample image is calculated by the semantic segmentation network, and the semantic segmentation network is iteratively trained based on the error value.

[0070] In one optional embodiment, the results of the semantic segmentation network are backpropagated, and the weights of the semantic segmentation network are corrected based on the error value between the predicted mask and the real mask corresponding to the same sample image, thereby training the semantic segmentation network.

[0071] The sample image is input into the semantic segmentation network, which performs semantic segmentation on the moving targets in the sample image. Training of the semantic segmentation network stops when the error between the predicted mask and the ground truth mask of the sample image is less than a preset threshold, which can be set by the user, such as 1% or 5%.

[0072] S203: Training the violent sorting detection network.

[0073] Please see Figure 5 , Figure 5 This is a schematic diagram of a specific embodiment of the violent sorting detection network provided by the present invention.

[0074] The violent sorting detection network in this embodiment employs a multilayer perceptron (MLP). The MLP comprises a cascaded input layer, a hidden unit, and an output layer. The hidden unit consists of three cascaded hidden layers.

[0075] Specifically, the steps for training the violent sorting detection network are as follows.

[0076] Please see Figure 6 , Figure 6 yes Figure 2 A flowchart illustrating a specific embodiment of step S203 in the provided violent sorting detection method.

[0077] S2031: Obtain the third training sample set.

[0078] Specifically, the third training sample set includes scene flow amplitude histogram sample data corresponding to multiple sets of moving targets. The amplitude range of the scene flow amplitude histogram sample data is [0, 500], and the spacing between the histograms in the scene flow amplitude histogram sample data is 50.

[0079] In one embodiment, multiple frames of images from a video stream are acquired, and the scene flow amplitude histogram corresponding to the moving targets contained in each video frame is calculated. Using the scene flow amplitude histogram as training data, information is labeled on each scene flow amplitude histogram to obtain the labeled behavior category for each scene flow amplitude histogram. The labeled behavior categories include two types: violent sorting and non-violent sorting.

[0080] S2032: The scene flow amplitude histogram sample data is analyzed by a violent sorting detection network to obtain the predicted category of the moving target.

[0081] Specifically, the violent sorting detection network predicts the probability value of a moving target belonging to the violent sorting category based on the scene flow amplitude histogram sample data, and determines the predicted category of the moving target based on the probability value. The predicted category includes both violent sorting and non-violent sorting categories.

[0082] If the probability value corresponding to the scene flow amplitude histogram sample data of the moving target exceeds the probability threshold, the predicted category of the moving target is determined to be the violent sorting category; if the probability value corresponding to the scene flow amplitude histogram sample data of the moving target does not exceed the probability threshold, the predicted category of the moving target is determined to be the non-violent sorting category.

[0083] S2033: The brute-force sorting detection network is trained iteratively based on the error between the predicted category and the labeled behavior category corresponding to the histogram sample data of the flow amplitude in the same scene.

[0084] Specifically, the error value between the predicted category and the labeled behavior category corresponding to the histogram sample data of the flow amplitude in the same scene is calculated by the brute-force sorting detection network, and the brute-force sorting detection network is iteratively trained based on the error value.

[0085] In one optional embodiment, the results of the violent sorting detection network are backpropagated, and the weights of the violent sorting detection network are corrected based on the error value between the predicted category and the labeled behavior category corresponding to the histogram sample data of the flow amplitude in the same scene, thereby achieving the training of the violent sorting detection network.

[0086] The scene flow amplitude histogram sample data is input into the violent sorting detection network, which analyzes the scene flow amplitude histogram sample data. When the error value between the predicted category and the labeled behavior category corresponding to the same scene flow amplitude histogram sample data is less than a preset threshold, the training of the violent sorting detection network is stopped. The preset threshold can be set by the user, such as 1% or 5%.

[0087] In this embodiment, the moving target can be a pedestrian, a delivery package, etc. In the following embodiment, a delivery package is used as an example of the moving target.

[0088] S204: Get the current video frame.

[0089] Specifically, image acquisition equipment is installed in the corresponding scene where images need to be captured. This image acquisition equipment is a binocular camera. The binocular camera is installed at an angle from the top in the corresponding scene, allowing it to capture images of the sorting station in the logistics sorting scene, thus obtaining clearer images containing express parcels. The monitoring footage from the binocular camera can cover the moving area of ​​the express parcels in the scene.

[0090] In one embodiment, before acquiring the current video frame, stereo calibration and correction are performed on the left and right cameras of the stereo camera to ensure that their optical axes are perfectly parallel. Specifically, Bouguet's epipolar correction algorithm is used to calibrate and correct the left and right cameras in the stereo camera.

[0091] In this embodiment, the optical axis of the right camera is corrected using the left camera as a reference, making it parallel to the optical axis of the left camera, thereby obtaining the intrinsic and extrinsic parameters of the binocular camera. In another embodiment, the intrinsic and extrinsic parameters of the binocular camera are adjusted to make the optical axis of the right camera parallel to the optical axis of the left camera. Images of the moving area of ​​the express package are then captured using the binocular camera.

[0092] In another embodiment, the optical axis of the left camera can be corrected by using the right camera in the binocular camera as a reference, so that the optical axis of the left camera is parallel to the optical axis of the right camera, thereby determining the intrinsic and extrinsic parameters of the binocular camera.

[0093] In a logistics sorting scenario involving express parcels, a top-mounted binocular camera captures images of the scene, resulting in the current video frame. This current video frame is a binocular image, meaning it includes both a left-eye and a right-eye image. Both the left and right-eye images contain the express parcels. The directly acquired right-eye image is corrected based on the binocular camera's intrinsic and extrinsic parameters to ensure that the optical axes of the left and right-eye images in the current video frame are parallel.

[0094] In another embodiment, the current video frame is acquired using a calibrated binocular camera. Specifically, the binocular camera, with adjusted intrinsic and extrinsic parameters, acquires images of a logistics sorting scene of express parcels to obtain the current video frame. The optical axes of the left and right images in the current video frame are parallel to each other. Both the left and right images in the current video frame contain express parcels.

[0095] S205: Based on the left and right eye images corresponding to the current video frame, obtain the disparity map of the current video frame.

[0096] Specifically, in order to obtain accurate image depth information, the disparity map of the current video frame is calculated using the left and right eye images of the current video frame, and then the depth information of the current video frame is calculated based on the intrinsic and extrinsic parameters of the binocular camera.

[0097] In one embodiment, in order to meet the real-time requirements and improve the detection speed of the disparity map of the current video frame, a semi-global block matching (SGBM) algorithm is used to obtain the disparity map of the current video frame based on the left and right eye images corresponding to the current video frame.

[0098] S206: In response to the fact that the current video frame is not the first frame image, an optical flow estimation network is used to generate the optical flow map corresponding to the current video frame based on the left eye image of the current video frame and the left eye images of the historical video frames adjacent to the current video frame.

[0099] Specifically, in this embodiment, the basis for determining whether the sorting of express parcels constitutes violent sorting is at least two video frames within the same video stream. Therefore, when determining the sorting behavior of express parcels in the current video frame, it is necessary to first determine whether the current video frame is the first frame of the video stream. If the current video frame is the first frame, after calculating the disparity map corresponding to the first frame, the second frame is directly acquired through a binocular camera. Then, based on the second frame and the first frame, the optical flow map corresponding to the second frame is determined.

[0100] In one specific embodiment, in response to the current video frame being a non-first frame image, the optical flow estimation network trained in step S201 generates an optical flow map corresponding to the current video frame based on the left-eye image of the current video frame and the left-eye images in adjacent historical video frames. The optical flow map represents the instantaneous velocity of each pixel as it moves from its coordinate position in an adjacent historical video frame to its coordinate position in the current video frame.

[0101] In another embodiment, an optical flow map corresponding to the current video frame is generated based on the right-eye image of the current video frame and the right-eye images of historical video frames adjacent to the current video frame.

[0102] S207: The left or right eye image in the historical video frames adjacent to the current video frame is segmented using a semantic segmentation network to determine the mask image of the current video frame.

[0103] Specifically, the semantic segmentation network trained in step S202 above segments the left-eye image in the current video frame and the left-eye images in adjacent historical video frames, respectively, to obtain the mask image corresponding to each video frame. The mask image is used to display the location and contour information of the target object, specifically the location and contour information of the express parcel area.

[0104] In this embodiment, the express parcel is the main object of motion analysis. Therefore, only the scene flow information of the express parcel can be calculated, and then the sorting behavior of the express parcel can be determined based on the scene flow information of the express parcel, thereby improving the calculation speed.

[0105] Based on binocular stereo imaging theory, as well as parallax and optical flow, it is known that the same target point or local target region is projected as a pixel or pixel region in two consecutive binocular images. The pixel grayscale of this point or pixel region remains unchanged between the two consecutive binocular images. Using this principle, reliable optical flow and parallax information of express delivery packages can be extracted by filtering image grayscale data. This reliable optical flow and parallax information can then be used to calculate the scene flow of a relatively sparse express delivery package region.

[0106] S208: Based on the mask image of the current video frame, extract the optical flow information of the target object from the optical flow map corresponding to the current video frame, and extract the disparity information of the target object from the disparity map corresponding to the current video frame.

[0107] Specifically, based on the location and contour information of the parcel region in the mask image of the current video frame, the optical flow information corresponding to the parcel region is extracted from the optical flow map of the current video frame, and used as the optical flow information of the parcel in the current video frame. The optical flow information of the parcel includes the movement distance of the parcel.

[0108] Based on the location and contour information of the parcel area in the mask image of the current video frame, extract the disparity information of the parcel area from the disparity map corresponding to the current video frame.

[0109] Based on the distance the express parcel moves from its position coordinates in a historical video frame adjacent to the current video frame to its current position coordinates, express parcels with larger movement ranges are selected as candidate targets from all express parcels.

[0110] S209: Determine whether the movement distance of the target object exceeds the preset distance.

[0111] Specifically, the movement distance of each express package in the current video frame is compared with the preset distance.

[0112] If the distance traveled by the express package exceeds the preset distance, proceed directly to step S210; if the distance traveled by the express package does not exceed the preset distance, proceed directly to step S211.

[0113] S210: Identify the target object as a candidate target.

[0114] Specifically, if the movement distance of a target object exceeds a preset distance, the target object is identified as a candidate target. If the movement distance of a parcel exceeds a preset distance, it indicates that the sorting of the parcel may be a violent sorting behavior, and the parcel area is identified as a candidate target.

[0115] S211: Remove the target object.

[0116] Specifically, if the movement distance of the target object does not exceed the preset distance, the target object is removed. If the movement distance of the express package does not exceed the preset distance, it indicates that the sorting of the express package is a non-violent sorting behavior, and the express package is removed and no further processing is carried out on the express package.

[0117] Please see Figure 7 , Figure 7 This is a schematic diagram of a specific embodiment of the violent sorting detection method provided by the present invention.

[0118] In this embodiment, the historical video frame is a video frame adjacent to the current video frame. The left and right view images of the historical video frame are represented as L, respectively. 0 R 0 The left and right view images of the current video frame are represented as L, respectively. 1 R 1 The disparity map of the current video frame is D. 1 The disparity map of historical video frames is D. 0 The historical video frames were acquired at time t0, and the current video frame was acquired at time t1; the optical flow graph corresponding to the current video frame is F. L ; This represents the region image of the candidate target in the left eye image of a historical video frame; SF s This represents the sparse scene flow of the candidate targets. Based on the theory of constant brightness between adjacent video frames, and combining the disparity maps of historical video frames, the disparity map of the current video frame, the optical flow map corresponding to the current video frame, and the region image of the candidate targets, the scene flow of the express parcel region corresponding to the candidate targets can be calculated.

[0119] S212: Based on the disparity information of the candidate target in the historical video frames adjacent to the current video frame, the regional image of the candidate target, and the device parameters, determine the position information of the candidate target at the first moment.

[0120] In one embodiment, based on the disparity map D of historical video frames 0 Region image of candidate targets in the left eye image of historical video frames. By combining the intrinsic and extrinsic parameters of the stereo camera that acquired historical and current video frames, the position coordinates (XYZ_t0) of the candidate target in the world coordinate system at the first moment can be calculated. The first moment is the acquisition time of the historical video frame adjacent to the current video frame.

[0121] S213: Based on the optical flow information of the candidate target, the parallax information of the candidate target in the current video frame, and the device parameters, determine the position information of the candidate target at the second time moment.

[0122] In one embodiment, based on the disparity map and optical flow map of the current video frame, as well as the intrinsic and extrinsic parameters of the stereo camera that acquired the historical and current video frames, the position coordinates XYZ_t1 of the candidate target in the world coordinate system at the second time point can be calculated. The second time point is the acquisition time of the current video frame. The position information is the coordinate position in the world coordinate system.

[0123] S214: Determine the position variable information of the candidate target based on the position information of the candidate target at the first time and the position information at the second time.

[0124] In one embodiment, the sparse scene flow SF of the candidate target region is obtained by subtracting the position information XYZ_t0 of the candidate target region at the first time step from the position information XYZ_t1 of the candidate target region at the second time step. s In this embodiment, sparse scene flow SF s This refers to positional variable information, which includes the direction of movement and the relative distance of movement.

[0125] In this embodiment, the disparity map of historical video frames, the disparity map of the current video frame, and the optical flow map are used to calculate the sparse scene flow of candidate targets. The scene flow represents the real motion of candidate targets in a real 3D scene, which can greatly solve the problem of difficulty in obtaining stable motion information of express parcels due to the principle that images captured by the camera are larger when they are closer and smaller when they are farther away.

[0126] Sparse scene flow represents the motion vector of a package area in a real 3D scene. Its amplitude is the magnitude of the motion distance, and its direction is the actual running direction of the package area in the 3D scene. When the package moves more vigorously, the amplitude of the sparse scene flow is larger, and when the package moves more gently, the amplitude of the sparse scene flow is smaller.

[0127] S215: Generate a relative distance histogram of the candidate target based on the relative motion distance of each pixel corresponding to the candidate target.

[0128] Specifically, the relative distance histogram corresponding to the express parcel area is calculated based on the amplitude of the sparse scene flow of the express parcel area in the current video frame. The amplitude range in the relative distance histogram is [0, 500], and the group interval of the histogram is 50.

[0129] S216: Based on the relative distance histogram of candidate targets, determine whether the sorting behavior of candidate targets involves violent sorting.

[0130] Specifically, a brute-force sorting detection network is used to determine the probability value of whether brute-force sorting occurs in the sorting behavior of candidate targets based on the relative distance histogram of target objects. If the probability value corresponding to a target object exceeds a preset value, then brute-force sorting is determined to occur. If the probability value corresponding to a target object does not exceed the preset value, then brute-force sorting is determined not to occur.

[0131] In one embodiment, to improve detection accuracy, the relative distance histogram of the candidate target in the current video frame can be determined based on the weighted sum of the relative distance histograms of a consecutive preset number of frames corresponding to the candidate target. Then, based on the weighted summed relative distance histogram, it can be determined whether the sorting behavior of the candidate target in the current video frame involves violent sorting. The weighting coefficient decreases as the time since the current video frame was captured increases, and the weighting coefficient increases as the time since the current video frame was captured increases.

[0132] In this embodiment, violent sorting detection is performed based on the sparse scene flow of candidate targets. Violent sorting behavior is detected by using the scene flow information of the express parcel area, which can avoid false detections caused by the violent movement of sorting personnel.

[0133] The violent sorting detection method provided in this embodiment includes: determining the disparity map and optical flow map corresponding to the current video frame based on the acquired current video frame and historical video frames before the current video frame; performing semantic segmentation on the historical video frames before the current video frame to determine the region image of the target object; determining the position variable information of the target object based on the region image of the target object, the disparity map and optical flow map corresponding to the current video frame; and determining whether violent sorting occurs in the current video frame based on the position variable information of the target object. This application determines the position variable information of the target object based on the disparity map, optical flow map and region image of the target object corresponding to the current video frame. By using the position variable information of the target object, the actual motion of the target object in a real 3D scene can be determined, which can greatly solve the problem of difficulty in obtaining stable target object motion information due to the principle of near-to-far image size in camera-acquired images, thereby improving the detection accuracy of the sorting behavior of the target object.

[0134] See Figure 8 , Figure 8 This is a schematic diagram of a framework of an embodiment of the violent sorting detection device provided by the present invention. This embodiment provides a violent sorting detection device 60, which includes an acquisition module 61, a segmentation module 62, an analysis module 63, and a processing module 64.

[0135] The acquisition module 61 is used to determine the disparity map and optical flow map corresponding to the current video frame based on the acquired current video frame and the historical video frames before the current video frame; the current video frame and the historical video frames are multi-view images captured by each lens in the multi-view camera at different times, and the multi-view images contain the target object.

[0136] The segmentation module 62 is used to perform semantic segmentation on historical video frames before the current video frame to determine the region image of the target object.

[0137] Analysis module 63 is used to determine the positional variable information of the target object based on the region image of the target object, the disparity map and optical flow map corresponding to the current video frame.

[0138] The processing module 64 is used to determine whether the sorting of target objects in the current video frame involves violent sorting based on the position variable information of the target object.

[0139] The violent sorting detection device provided in this embodiment determines the positional variable information of the target object based on the disparity map, optical flow map and regional image of the target object corresponding to the current video frame. The positional variable information of the target object can determine the real motion of the target object in the real three-dimensional scene, which can greatly solve the problem of difficulty in obtaining stable target object motion information due to the principle of near objects appearing larger and distant objects appearing smaller, thereby improving the detection accuracy of sorting target object behavior.

[0140] Please see Figure 9 , Figure 9 This is a schematic diagram of a terminal embodiment of this application. The terminal 80 includes a memory 81 and a processor 82 coupled to each other. The processor 82 is used to execute program instructions stored in the memory 81 to implement the steps of any of the above-described violent sorting detection method embodiments. In a specific implementation scenario, the terminal 80 may include, but is not limited to, a microcomputer or a server. In addition, the terminal 80 may also include mobile devices such as laptops and tablets, which are not limited here.

[0141] Specifically, processor 82 controls itself and memory 81 to implement the steps of any of the above-described violent sorting detection method embodiments. Processor 82 can also be referred to as a CPU (Central Processing Unit). Processor 82 may be an integrated circuit chip with signal processing capabilities. Processor 82 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 82 can be implemented using integrated circuit chips.

[0142] Please see Figure 10 , Figure 10 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor. The program instructions 901 are used to implement the steps of any of the above-described embodiments of the violent sorting detection method.

[0143] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0144] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0146] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0147] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0149] The above are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for detecting violent sorting, characterized in that, The violent sorting detection method includes: Based on the acquired current video frame and the historical video frames preceding the current video frame, the disparity map and optical flow map corresponding to the current video frame are determined; the current video frame and the historical video frames are multi-view images captured by each lens of the multi-view camera at different times, and the multi-view images contain the target object; Semantic segmentation is performed on the historical video frames preceding the current video frame to determine the region image of the target object; Based on the region image of the target object, the disparity map corresponding to the current video frame, and the optical flow map, the positional variable information of the target object is determined; the positional variable information includes the direction of motion and the relative distance of motion. Based on the location variable information of the target object, determine whether the sorting behavior of the target object in the current video frame involves violent sorting; The step of determining whether the sorting of the target object in the current video frame involves violent sorting based on the location variable information of the target object includes: Based on the relative motion distance of each pixel corresponding to the target object, a relative distance histogram of the target object is generated; Based on the relative distance histogram of the target object, determine whether the sorting behavior of the target object in the current video frame involves violent sorting.

2. The violent sorting detection method according to claim 1, characterized in that, The multi-view image includes at least a left-view image and a right-view image; both the left-view image and the right-view image contain the target object; The step of determining the disparity map and optical flow map corresponding to the current video frame based on the acquired current video frame and the historical video frames preceding the current video frame includes: Get the current video frame; In response to the fact that the current video frame is not the first frame image, a disparity map of the current video frame is obtained based on the left eye image and the right eye image corresponding to the current video frame; An optical flow map corresponding to the current video frame is generated based on the left eye image of the current video frame and the left eye images of the historical video frames adjacent to the current video frame; or, an optical flow map of the current video frame is generated based on the right eye image of the current video frame and the right eye images of the historical video frames adjacent to the current video frame.

3. The violent sorting detection method according to claim 2, characterized in that, The step of obtaining the disparity map of the current video frame based on the left-eye image and the right-eye image corresponding to the current video frame includes: A semi-global matching algorithm is used to obtain the disparity map of the current video frame based on the left and right eye images corresponding to the current video frame.

4. The violent sorting detection method according to claim 2, characterized in that, The step of semantically segmenting the historical video frames preceding the current video frame to determine the region image of the target object includes: A semantic segmentation network is used to segment the left or right image in the historical video frames adjacent to the current video frame to determine the region image of the target object.

5. The violent sorting detection method according to claim 2, characterized in that, The determination of the positional variable information of the target object based on the region image of the target object, the disparity map corresponding to the current video frame, and the optical flow map includes: Based on the region image of the target object, the optical flow information of the target object is extracted from the optical flow map corresponding to the current video frame, and the disparity information of the target object is extracted from the disparity map corresponding to the current video frame; Based on the disparity information of the target object in the historical video frames adjacent to the current video frame and the region image of the target object, the position information of the target object at the first moment is determined; the first moment is the acquisition time of the historical video frame adjacent to the current video frame; Based on the optical flow information of the target object and the parallax information of the target object in the current video frame, the position information of the target object at a second time moment is determined; the second time moment is later than the first time moment; the second time moment is the acquisition time of the current video frame; Based on the position information of the target object at the first time and the position information at the second time, the position variable information of the target object is determined.

6. The violent sorting detection method according to claim 5, characterized in that, The optical flow information of the target object includes the movement distance of the target object; The step of determining the position information of the target object at the first moment based on the disparity information of the target object in the historical video frames adjacent to the current video frame and the region image of the target object, further includes: Determine whether the movement distance of the target object exceeds a preset distance; If the movement distance of the target object exceeds the preset distance, the target object is identified as a candidate target; If the movement distance of the target object does not exceed the preset distance, the target object is removed. The step of determining the position information of the target object at a first moment based on the disparity information of the target object in the historical video frames adjacent to the current video frame and the region image of the target object includes: Based on the disparity information of the candidate target in the historical video frames adjacent to the current video frame, the regional image of the candidate target, and device parameters, the position information of the candidate target at the first moment is determined; Determining the position information of the target object at the second time moment based on the optical flow information of the target object and the disparity information of the target object in the current video frame includes: Based on the optical flow information of the candidate target, the parallax information of the candidate target in the current video frame, and the device parameters, the position information of the candidate target at the second time moment is determined; the position information is the coordinate position in the world coordinate system.

7. The violent sorting detection method according to claim 1, characterized in that, The step of determining whether the sorting of the target object in the current video frame involves brute force based on the relative distance histogram of the target object includes: A brute-force sorting detection network is used to determine the probability value of the target object being subjected to brute-force sorting based on the relative distance histogram of the target object. The brute-force sorting detection network is trained by a sample histogram obtained by weighting and summing the relative distance histograms corresponding to sample images of a consecutive preset number of frames. The sample histogram is associated with labeled behavior categories, which include brute-force sorting and non-brute-force sorting. If the probability value corresponding to the target object exceeds a preset value, it is determined that the sorting of the target object in the current video frame involves violent sorting. If the probability value corresponding to the target object does not exceed the preset value, it is determined that the sorting behavior of the target object in the current video frame does not involve violent sorting.

8. A violent sorting detection device, characterized in that, The violent sorting detection device includes: The acquisition module is used to determine the disparity map and optical flow map corresponding to the current video frame based on the acquired current video frame and the historical video frames preceding the current video frame; the current video frame and the historical video frames are multi-view images captured by each lens of the multi-view camera at different times, and the multi-view images contain the target object; The segmentation module is used to perform semantic segmentation on the historical video frames preceding the current video frame to determine the region image of the target object; The analysis module is used to determine the positional variable information of the target object based on the region image of the target object, the disparity map corresponding to the current video frame, and the optical flow map; the positional variable information includes the direction of motion and the relative distance of motion. The processing module is used to determine whether the sorting behavior of the target object in the current video frame is violent sorting based on the position variable information of the target object; it is also used to generate a relative distance histogram of the target object based on the relative motion distance of each pixel corresponding to the target object; and to determine whether the sorting behavior of the target object in the current video frame is violent sorting based on the relative distance histogram of the target object.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, the processor being used to execute program data to implement the steps in the violent sorting detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the violent sorting detection method as described in any one of claims 1 to 7.

Images