Leftover item detection method, apparatus, device, medium, and program product

Abstract

The present disclosure provides a left-behind article detection method, which can be applied to the field of artificial intelligence technology. The left-behind article detection method comprises: receiving video stream information; receiving video stream information; determining whether a personnel target appears in the video stream information; in the case where the personnel target appears in the video stream information, detecting whether a left-behind article exists; in the case where the personnel target appears in the video stream information, continuously sending the video stream in the video stream information from the appearance of the personnel target to a remote server, so as to monitor the motion track of the personnel target by the remote server; and in the case where it is detected that the left-behind article exists, obtaining the motion track from the remote server. The present disclosure also provides a left-behind article detection device, equipment, storage medium and program product.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and specifically to a method, apparatus, device, medium, and program product for detecting abandoned items. Background Technology

[0002] Cameras deployed at operational outlets and other locations are used to monitor environmental security and have accumulated a large amount of image and video data resources. With the improvement of edge intelligence technology and computer vision technology, it is possible to consider using computer vision technology to automatically detect objects in video images, realize functions such as left-behind item detection and abnormal item detection, thereby improving the efficiency of outlet operations and internal operations.

[0003] Typically, the detection process for abandoned items is not conducted at operational sites, but rather at remote locations outside of them. Operational sites, however, utilize dedicated networks with limited bandwidth and computing power. Therefore, how to effectively identify abandoned items at operational sites while maintaining sufficient dedicated network bandwidth is a pressing technical challenge for those skilled in the art. Summary of the Invention

[0004] In view of the above problems, this disclosure provides legacy item detection methods, apparatus, equipment, media, and program products that improve the efficiency of legacy item detection and reduce the computational overhead of local legacy item detection.

[0005] According to a first aspect of this disclosure, a method for detecting abandoned items is provided, the method being applied to a branch server, the method comprising: receiving video stream information; determining whether a person or target appears in the video stream information; if a person or target appears in the video stream information, detecting whether abandoned items exist; if a person or target appears in the video stream information, continuously uploading the video stream from which the person or target appears to a remote server, so that the remote server monitors the movement trajectory of the person or target; and if the abandoned items are detected, acquiring the movement trajectory from the remote server.

[0006] According to an embodiment of this disclosure, the video stream information includes multiple video frame images. Determining whether a person or object appears in the video stream information includes: acquiring a reference image, where the reference image is an image of the monitored area corresponding to the video stream information where the person or object does not exist and / or the abandoned item does not exist; calculating the difference between the reference image and the video frame images based on a frame difference algorithm to obtain a first difference; and determining that the person or object appears in the video stream information if the first difference is outside a preset error range.

[0007] According to embodiments of this disclosure, the plurality of video frame images includes at least a first video frame image and a second video frame image, the first video frame image and the second video frame image are sequentially continuous, the first video frame image is the last video frame image in which the target customer is within the monitoring area, and the detection of whether there is a leftover item includes: determining whether the target person has left the monitoring area; if the target person has left the monitoring area, acquiring the second video frame image; acquiring the comparison image; calculating the difference between the comparison image and the second video frame image based on a frame difference algorithm to obtain a second difference; and if the second difference is outside a preset error range, determining that the leftover item appears in the video stream information.

[0008] According to an embodiment of this disclosure, the step of calculating the difference between the control image and the video frame image based on the frame difference algorithm to obtain a first difference includes: dividing the video frame image and the control image into a preset number of sub-unit regions according to a preset division method, wherein each sub-unit region includes a unit identifier; and calculating the difference between the sub-unit regions in the control image and the sub-unit regions in the video frame image one by one based on the frame difference algorithm according to the correspondence of the unit identifiers to obtain the first sub-difference.

[0009] According to an embodiment of this disclosure, determining that the person target appears in the video stream information when the first difference is outside a preset error range includes: determining that the person target appears in the video stream information when the first sub-difference is outside a preset error range, and stopping the calculation of the first sub-difference of the remaining sub-unit regions.

[0010] According to an embodiment of this disclosure, the step of calculating the difference between the control image and the second video frame image based on the frame difference algorithm to obtain a second difference includes: dividing the second video frame image and the control image into a preset number of sub-unit regions according to a preset division method, wherein each sub-unit region includes a unit identifier; and calculating the difference between each sub-unit region in the control image and the sub-unit region in the video frame image based on the frame difference algorithm according to the correspondence of the unit identifiers to obtain a second sub-difference.

[0011] According to an embodiment of this disclosure, determining that the abandoned item appears in the video stream information when the second difference is outside a preset error range includes: determining that the person target appears in the video stream information when the second sub-difference is outside a preset error range, and stopping the calculation of the second sub-difference for the remaining sub-unit regions.

[0012] According to an embodiment of this disclosure, when a person target appears in the video stream information, after detecting whether there is a left-behind item, the method further includes: issuing a left-behind item alarm command when a left-behind item is detected.

[0013] According to an embodiment of this disclosure, when a person target appears in the video stream information, after detecting whether there are any abandoned items, the method further includes: if the abandoned items are detected, identifying the abandoned items based on a preset abandoned item recognition model and outputting an abandoned item classification.

[0014] According to an embodiment of this disclosure, the training method for the preset abandoned item recognition model includes: acquiring training samples and sample labels, wherein the training samples include images of historical abandoned items and the sample labels include item categories; and using the images of historical abandoned items and the item categories as input data to train the model until a preset number of training iterations is reached.

[0015] A second aspect of this disclosure provides a device for detecting abandoned items, the device being installed in a branch server. The device includes: a video receiving module for receiving video stream information; a personnel target judgment module for determining whether a personnel target appears in the video stream information; an abandoned item detection module for detecting the presence of abandoned items when a personnel target appears in the video stream information; a video uploading module for continuously uploading the video stream from which the personnel target appears to a remote server when a personnel target appears in the video stream information, so that the remote server can monitor the movement trajectory of the personnel target; and a movement trajectory acquisition module for acquiring the movement trajectory from the remote server when the presence of the abandoned item is detected.

[0016] According to an embodiment of this disclosure, the personnel target determination module is further configured to acquire a reference image, wherein the reference image is an image in which the personnel target and / or the abandoned items do not exist in the monitoring area corresponding to the video stream information; calculate the difference between the reference image and the video frame image based on a frame difference algorithm to obtain a first difference; and determine that the personnel target appears in the video stream information when the first difference is outside a preset error range.

[0017] According to embodiments of this disclosure, the plurality of video frame images includes at least a first video frame image and a second video frame image, the first video frame image and the second video frame image are sequentially continuous, the first video frame image is the last video frame image in which the target customer is within the monitoring area, and the abandoned item detection module is further configured to determine whether the target person has left the monitoring area; if the target person has left the monitoring area, acquire the second video frame image; acquire the comparison image; calculate the difference between the comparison image and the second video frame image based on a frame difference algorithm to obtain a second difference; and if the second difference is outside a preset error range, determine that the abandoned item appears in the video stream information.

[0018] According to an embodiment of this disclosure, the personnel target judgment module is further configured to divide the video frame image and the control image into a preset number of sub-unit regions according to a preset division method, wherein the sub-unit region includes a unit identifier; and to calculate the difference between the sub-unit regions in the control image and the sub-unit regions in the video frame image one by one based on the frame difference algorithm according to the correspondence of the unit identifiers, so as to obtain a first sub-difference value.

[0019] According to an embodiment of this disclosure, the personnel target determination module is further configured to determine that the personnel target appears in the video stream information when the first sub-difference value is outside a preset error range, and to stop calculating the first sub-difference value of the remaining sub-unit regions.

[0020] According to an embodiment of this disclosure, the abandoned item detection module is further configured to divide the second video frame image and the control image into a preset number of sub-unit regions according to a preset division method, wherein the sub-unit region includes a unit identifier; and to calculate the difference between the sub-unit regions in the control image and the sub-unit regions in the video frame image one by one based on the frame difference algorithm according to the correspondence of the unit identifiers, so as to obtain a second sub-difference value.

[0021] According to an embodiment of this disclosure, the abandoned item detection module is further configured to determine that the person target appears in the video stream information when the second sub-difference value is outside a preset error range, and to stop calculating the second sub-difference value of the remaining sub-unit regions.

[0022] According to an embodiment of this disclosure, the device further includes: an alarm module, configured to detect whether there is a left-behind item when a person target appears in the video stream information, and further includes: issuing a left-behind item alarm command when the left-behind item is detected.

[0023] According to an embodiment of this disclosure, the device further includes: a leftover items classification module, used to identify the leftover items based on a preset leftover items recognition model and output a leftover items classification when the presence of the leftover items is detected.

[0024] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the aforementioned method for detecting abandoned items.

[0025] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the aforementioned method for detecting abandoned items.

[0026] A fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for detecting abandoned items.

[0027] In the embodiments of this disclosure, compared with the prior art, this disclosure includes at least the following advantages: 1. By monitoring the movement trajectory of personnel targets within the monitoring area before starting the detection of abandoned items, the computational cost of abandoned item detection can be reduced, and the presence of abandoned items can be identified more accurately. 2. By using time as the detection dimension, abandoned items of personnel targets are detected, assisting in improving the operational capabilities of the network. 3. The target tracking, which has a high computational cost, is placed on a remote server for identification, making up for the shortcomings of the limited computing power of local servers. Attached Figure Description

[0028] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0029] Figure 1 The diagram illustrates an application scenario of the method for detecting abandoned items according to an embodiment of the present disclosure.

[0030] Figure 2 A flowchart illustrating a method for detecting abandoned items according to an embodiment of the present disclosure is shown schematically.

[0031] Figure 3 A flowchart illustrating a method for determining the movement of a person target according to an embodiment of the present disclosure is shown schematically.

[0032] Figure 4 A flowchart illustrating a method for detecting abandoned items according to an embodiment of the present disclosure is shown schematically.

[0033] Figure 5A flowchart illustrating another method for determining the movement of a person target according to an embodiment of the present disclosure is shown schematically.

[0034] Figure 6 A flowchart illustrating another method for determining the movement of a person target according to an embodiment of the present disclosure is shown schematically.

[0035] Figure 7 A flowchart illustrating another method for detecting abandoned items according to an embodiment of this disclosure is shown schematically.

[0036] Figure 8 A flowchart illustrating another method for detecting abandoned items according to an embodiment of this disclosure is shown schematically.

[0037] Figure 9 A flowchart illustrating a method for detecting abandoned items according to an embodiment of the present disclosure is shown schematically.

[0038] Figure 10 The diagram illustrates a method for training a legacy item recognition model according to an embodiment of the present disclosure.

[0039] Figure 11 A schematic block diagram of a legacy item detection apparatus according to an embodiment of the present disclosure is shown.

[0040] Figure 12 A block diagram schematically illustrates an electronic device suitable for implementing a method for detecting abandoned items according to an embodiment of the present disclosure. Detailed Implementation

[0041] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0042] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0043] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0044] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0045] Before explaining the embodiments of this disclosure in detail, the key technologies involved in this application will be described one by one:

[0046] YOLO: An object detection algorithm used to find one or more classes of objects of interest in an image and determine the location of the object.

[0047] Faster R-CNN: An object detection algorithm used to find one or more classes of objects of interest in an image and determine the location of the object.

[0048] OpenCV: A cross-platform real-time image processing tool.

[0049] This disclosure provides a method for detecting abandoned items, applied to a branch server. The method includes: receiving video stream information; determining whether a person or target appears in the video stream information; if a person or target appears in the video stream information, detecting whether abandoned items exist; if a person or target appears in the video stream information, continuously uploading the video stream from which the person or target appears to a remote server so that the remote server can monitor the movement trajectory of the person or target; and if abandoned items are detected, acquiring the movement trajectory from the remote server.

[0050] In the embodiments of this disclosure, compared with the prior art, this disclosure includes at least the following advantages: 1. By monitoring the movement trajectory of personnel targets within the monitoring area before starting the detection of abandoned items, the computational cost of abandoned item detection can be reduced, and the presence of abandoned items can be identified more accurately. 2. By using time as the detection dimension, abandoned items of personnel targets are detected, assisting in improving the operational capabilities of the network. 3. The target tracking, which has a high computational cost, is placed on a remote server for identification, making up for the shortcomings of the limited computing power of local servers.

[0051] Figure 1 The diagram illustrates an application scenario of the method for detecting abandoned items according to an embodiment of the present disclosure.

[0052] like Figure 1As shown, application scenario 100 according to this embodiment may include a video capture device 101, a branch server 102, a network 103, and a remote server 104. The network 103 serves as a medium for providing a communication link between the branch server 102 and the remote server 104. The network 103 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0053] The video capture device 101 can collect environmental and personnel information at the site and upload it to the site server 102.

[0054] Remote server 104 can be a server that provides various services, such as a backend management server that supports the website browsed by the user using site server 102 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to site server 102.

[0055] It should be noted that the abandoned item detection method provided in this embodiment can generally be executed by the branch server 102. Correspondingly, the abandoned item detection device provided in this embodiment can generally be located in the branch server 102. The abandoned item detection method provided in this embodiment can also be executed by a server or server cluster that is different from the branch server 102 and capable of communicating with the branch server 102 and / or the remote server 104. Correspondingly, the abandoned item detection device provided in this embodiment can also be located in a server or server cluster that is different from the branch server 102 and capable of communicating with the branch server 102 and / or the remote server 104.

[0056] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0057] The following will be based on Figure 1 The described scene, through Figures 2 to 10 The method for detecting abandoned items according to the disclosed embodiments will be described in detail.

[0058] Figure 2 A flowchart illustrating a method for detecting abandoned items according to an embodiment of the present disclosure is shown schematically.

[0059] like Figure 2 As shown, the abandoned item detection method of this embodiment includes operations S210 to S250, and the abandoned item detection method can be executed by the site server 102.

[0060] In operation S210, video stream information is received. This video stream information includes multiple video frame images.

[0061] The detection of abandoned items is conducted within a branch office service environment. Video capture device 101 collects video stream information from this environment. This video stream typically includes at least one customer (the target person for subsequent image processing). For example, in the detection process, the video stream from the surveillance camera facing the bank branch teller counter is used as the input stream, processed into video image frames for detection, with the timeline as the dimension for abandoned item detection. This video stream information is then sent to the branch server 102 for further processing. Considering the low bandwidth of the dedicated network used by the branch and the limited computing power of the branch server 102, the entire abandoned item detection and identification process can be executed on different servers.

[0062] Specifically, considering that the network server 102 has poor data processing capabilities and the remote server 104 has strong data processing capabilities, the developers can deploy logic to detect "moving targets" and "leftover items" in the surveillance video on the network server 102, and deploy logic to detect "moving target trajectories" on the remote server 104, taking into account bandwidth and computing power.

[0063] In operation S220, it is determined whether a human target appears in the video stream information.

[0064] Specifically, the frame difference algorithm running on the network server 102 can be used to detect whether there is a moving target (that is, to detect whether the person is moving).

[0065] In operation S230, if a person target appears in the video stream information, it is detected whether there are any leftover items.

[0066] Specifically, the frame difference algorithm running on the branch server 102 can be used to detect whether there are any leftover items.

[0067] According to an embodiment of this disclosure, when a person target appears in the video stream information, after detecting whether there is a left-behind item, the method further includes: issuing a left-behind item alarm command when a left-behind item is detected.

[0068] For example, if lost or found items are discovered, an alarm can be triggered to notify the relevant personnel for handling.

[0069] In operation S240, if a person target appears in the video stream information, the video stream from which the person target appears is continuously uploaded to a remote server so that the remote server can monitor the movement trajectory of the person target.

[0070] Generally, moving targets appearing in bank branches are considered personnel targets. Therefore, moving targets are equated with personnel targets. When a personnel target is detected, the video stream showing the personnel target is sent to the remote server 104. Specifically, the video processing process can involve the branch server 102 converting the received video stream information into an image program, encapsulating it, and using independent camera video streams or recorded videos from the actual application scenario of object recognition as input, providing frame interval parameters and resolution parameters for video image frame capture, and outputting the converted video frames. For example, in video processing, OpenCV-like video processing tools are used to process the video data, using the video frame interval parameter as the video frame counting interval frequency, and continuously reading video frames. When the resolution of the acquired video frame is higher than or equal to the resolution parameter, it is saved to a specified path.

[0071] The trajectory of a person is tracked using a trajectory tracking algorithm. This algorithm combines the YOLO v5 target detection algorithm with a target tracking algorithm, and further integrates a person identification re-identification algorithm.

[0072] In operation S250, if the presence of the abandoned item is detected, the movement trajectory is obtained from the remote server.

[0073] For example, determine whether the person is still within the monitoring range. If the person is still within the monitoring range, notify the lobby manager of the person's location information so that the lobby manager can return the lost item directly to the owner. If the person has left the monitoring range, trace back the counter location and corresponding time period where their historical trajectory appeared. Determine if the owner is a customer who used that counter during that time period, obtain the customer's contact information through the dispatch business system, and contact them to retrieve the lost item.

[0074] In the embodiments of this disclosure, compared with the prior art, this disclosure includes at least the following advantages: 1. By monitoring the movement trajectory of personnel targets within the monitoring area before starting the detection of abandoned items, the computational cost of abandoned item detection can be reduced, and the presence of abandoned items can be identified more accurately. 2. By using time as the detection dimension, abandoned items of personnel targets are detected, assisting in improving the operational capabilities of the network. 3. The target tracking, which has a high computational cost, is placed on a remote server for identification, making up for the shortcomings of the limited computing power of local servers.

[0075] The method for determining the movement of personnel targets according to embodiments of this disclosure will be described in detail below:

[0076] Figure 3 A flowchart illustrating a method for determining the movement of a person target according to an embodiment of the present disclosure is shown schematically.

[0077] like Figure 3 As shown, the personnel target movement determination method of this embodiment includes operations S310 to S330. Operations S310 to S330 can at least partially perform the above-described operation S220.

[0078] In operation S310, a comparison image is acquired, which is an image in which the person target and / or the abandoned item does not exist in the monitoring area corresponding to the video stream information.

[0079] For example, the comparison image mentioned above can be the first frame of the surveillance video at the start of business on the same day, in which the person or / or the abandoned items are not present.

[0080] In operation S320, the difference between the comparison image and the video frame image is calculated based on the frame difference algorithm to obtain the first difference.

[0081] In operation S330, if the first difference is outside the preset error range, it is determined that the person target appears in the video stream information.

[0082] Specifically, when an intrusive object appears in the monitored scene, there will be a relatively obvious difference between the frame images. The brightness difference obtained by subtracting the comparison image from the video frame image is the first difference value. If the absolute value of this difference value exceeds a threshold, it is determined that an intrusive object has appeared in the image.

[0083] In the embodiments of this disclosure, a frame difference algorithm with low overhead is used on the local server. By identifying the differences between different images, it is determined whether there is a scene with people present, which reduces the computing power overhead and fits the actual scene.

[0084] Figure 4 A flowchart illustrating a method for detecting abandoned items according to an embodiment of the present disclosure is shown schematically.

[0085] like Figure 4 As shown, another method for detecting items left behind by personnel in this embodiment includes operations S410 to S450. Operations S410 to S450 can at least partially perform the above-described operation S230.

[0086] According to embodiments of this disclosure, the plurality of video frame images includes at least a first video frame image and a second video frame image, the first video frame image and the second video frame image are sequentially consecutive, and the first video frame image is the last video frame image in which the target customer is within the monitoring area.

[0087] In operation S410, it is determined whether the personnel target has left the monitoring area.

[0088] Specifically, determining whether a person has left the monitored area can be achieved by receiving a command from the remote server 104 to leave the monitored area. This command includes a corresponding video frame image (or, of course, the time of the corresponding video frame image). Alternatively, the network server 102 can calculate the difference between video frames frame by frame using a frame difference algorithm. If the difference between the previous video frame image and the next video frame image is too large, then these two images are the first video frame and the second video frame, respectively.

[0089] Understandably, the interval between different frames can be a large value to ensure that there are no intermediate actions between different frames.

[0090] In operation S420, when the person target leaves the monitoring area, the second video frame image is acquired.

[0091] In operation S430, the comparison image is acquired.

[0092] In operation S440, the difference between the comparison image and the second video frame image is calculated based on the frame difference algorithm to obtain the second difference.

[0093] In operation S450, if the second difference is outside the preset error range, it is determined that the abandoned item appears in the video stream information.

[0094] Specifically, when an intrusive object appears in the monitored scene, there will be a significant difference between the frame images. The brightness difference obtained by subtracting the comparison image from the second video frame image is the second difference value. If the absolute value of this difference value exceeds a threshold, it is determined that an intrusive object has appeared in the image.

[0095] In the embodiments of this disclosure, a frame difference algorithm with low overhead is used on the local server. By identifying the differences between different images, it is determined whether there is a scene where leftover items appear, which reduces the computing power overhead and fits the actual scene.

[0096] In the process of calculating the first and second differences using the frame difference algorithm, the images to be used in the calculation can be further segmented to reduce the granularity of the calculation.

[0097] Figure 5 A flowchart illustrating another method for determining the movement of a person target according to an embodiment of the present disclosure is shown schematically.

[0098] like Figure 5 As shown, another method for determining the movement of personnel targets in this embodiment includes operations S510 to S520. Operations S510 to S520 can at least partially perform the above-described operation S320.

[0099] In operation S510, the video frame image and the comparison image are divided into a preset number of sub-unit regions according to a preset division method, wherein the sub-unit regions include unit identifiers.

[0100] In operation S520, according to the correspondence of the unit identifiers, the difference between the sub-unit regions in the comparison image and the sub-unit regions in the video frame image is calculated one by one based on the frame difference algorithm to obtain the first sub-difference value.

[0101] It should be noted that the above division of sub-unit regions can be of equal size or unequal size; that is, the different sub-unit regions can be of equal or unequal size.

[0102] Understandably, since the size of the sub-unit region can be freely defined, in each frame of video image, the sub-unit region involved in the person target can be only one or multiple. The aforementioned first sub-difference value has a one-to-one correspondence with the sub-unit region, with one sub-unit region corresponding to one first sub-difference value.

[0103] Figure 6 A flowchart illustrating another method for determining the movement of a person target according to an embodiment of the present disclosure is shown schematically.

[0104] like Figure 4 As shown, another method for determining personnel movement in this embodiment includes operation S610. This operation S610 can at least partially perform the above-described operation S330.

[0105] In operation S610, if the first sub-difference value is outside the preset error range, it is determined that the person target appears in the video stream information, and the calculation of the first sub-difference value of the remaining sub-unit regions is stopped.

[0106] It is understandable that the calculation of the first sub-difference can be performed one by one. When a certain first sub-difference is too large, it can be determined that a person target has appeared in the sub-unit area corresponding to the first sub-difference.

[0107] In the embodiments of this disclosure, the granularity of comparison between different frame images is reduced by dividing the image into sub-unit regions. When monitoring personnel movement, it is not necessary to calculate the entire image, thus partially freeing up the computing power of the computer.

[0108] Figure 7 A flowchart illustrating another method for detecting abandoned items according to an embodiment of this disclosure is shown schematically.

[0109] like Figure 7 As shown, another method for detecting items left behind by personnel in this embodiment includes operations S710 to S720. Operations S710 to S720 can at least partially perform the above-described operation S440.

[0110] In operation S710, the second video frame image and the comparison image are divided into a preset number of sub-unit regions according to a preset division method, wherein the sub-unit regions include unit identifiers.

[0111] In operation S720, according to the correspondence of the unit identifiers, the difference between the sub-unit regions in the comparison image and the sub-unit regions in the video frame image is calculated one by one based on the frame difference algorithm to obtain the second sub-difference value.

[0112] Understandably, since the size of the sub-unit region can be freely defined, the sub-unit region involved in the remaining item in each frame of video image can be either just one or multiple. The aforementioned second sub-difference value has a one-to-one correspondence with the sub-unit region; one sub-unit region corresponds to one second sub-difference value.

[0113] Figure 8 A flowchart illustrating another method for detecting abandoned items according to an embodiment of this disclosure is shown schematically.

[0114] like Figure 8 As shown, another method for detecting items left behind by personnel in this embodiment includes operation S810. This operation S810 can at least partially perform the above-described operation S450.

[0115] In operation S810, if the second sub-difference value is outside the preset error range, it is determined that the person target appears in the video stream information, and the calculation of the second sub-difference value of the remaining sub-unit regions is stopped.

[0116] Understandably, the calculation of the second sub-difference can be performed one by one. When a second sub-difference is too large, it can be determined that there are leftover items in the sub-unit area corresponding to that second sub-difference.

[0117] In the embodiments of this disclosure, the granularity of comparison between different frame images is reduced by dividing the image into sub-unit regions. When monitoring abandoned items, it is not necessary to calculate the entire image, thus partially freeing up the computing power of the computer.

[0118] Figure 9 A flowchart illustrating a method for detecting abandoned items according to an embodiment of the present disclosure is shown schematically.

[0119] like Figure 9 As shown, the method for detecting abandoned items in this embodiment, in addition to the operations S210 to S250 described above, also includes operation S910. Operation S910 is performed after operation S230.

[0120] In operation S910, when the presence of the abandoned item is detected, the abandoned item is identified based on a preset abandoned item identification model, and the abandoned item classification is output.

[0121] Specifically, the abandoned item identification model can be set on the branch server 102 or on the remote server 104. If the abandoned item identification model is set on the remote server 104, then it is only necessary to transmit the above-mentioned second video frame image to the remote server 104.

[0122] Figure 10 The diagram illustrates a method for training a legacy item recognition model according to an embodiment of the present disclosure.

[0123] like Figure 10 As shown, the training method for the abandoned item recognition model in this embodiment includes operations S1010 to S1020.

[0124] In operation S1010, training samples and sample labels are acquired. The training samples include images of historical abandoned items, and the sample labels include item categories.

[0125] In operation S1020, the images of the historical relics and the categories of the relics are used as input data to train the model until the preset number of training iterations is reached.

[0126] Specifically, the process involves acquiring surveillance video data of the scene where items were left behind, converting the video into image frames for output, and then using an image annotation tool to label the image frames with the categories of items to be detected, according to data annotation guidelines. The item detection self-training module is then run, providing an interface on the front end. Users input the acquired image frames and corresponding annotation data from "video data processing" and "data annotation," and the interface provides data analysis, outputting the amount of training data, item categories, and annotation quantity. When these three indicators meet the training requirements, semi-autonomous model retraining is performed. The interface provides functions for automatically separating the test and training sets, specifying the types and number of training items, the number of training iterations, the number of data samples captured in one training iteration, and customizing the model output path. Furthermore, it provides training parameter definition guidelines based on data conditions and usage scenarios. The process involves several steps: 1) After determining the training parameters, select the required baseline model on the interface, input the model parameters and training parameters, and automatically retrain the model with a single click. 2) After the model completes training, the interface displays the training results, including the number of training iterations, the number of test set images, the number of labels in the test set images, precision, and recall. When precision and recall meet the application requirements, the inference model is output. 3) For item detection applications, an engineering encapsulation framework for item detection is provided. The inference model obtained through "item detection self-training" can be applied to this framework and directly deployed in actual business applications. The framework's processing flow includes processing the RTSP stream to the server, converting the monitoring video stream into video frames, and then connecting it to the item detection model obtained through "model self-training."

[0127] Based on the above-described method for detecting abandoned items, this disclosure also provides a device for detecting abandoned items. The following will be combined with... Figure 11 The device is described in detail.

[0128] Figure 11 A schematic block diagram of a legacy item detection apparatus according to an embodiment of the present disclosure is shown.

[0129] like Figure 11 As shown, the abandoned item detection device 1100 of this embodiment includes a video receiving module 1110, a personnel target judgment module 1120, an abandoned item detection module 1130, a video uploading module 1140, and a motion trajectory acquisition module 1150.

[0130] The video receiving module 1110 is used to receive video stream information. In one embodiment, the video receiving module 1110 can be used to perform the operation S210 described above, which will not be repeated here.

[0131] The personnel target determination module 1120 is used to determine whether a personnel target appears in the video stream information. In one embodiment, the personnel target determination module 1120 can be used to perform the operation S220 described above, which will not be repeated here.

[0132] The abandoned object detection module 1130 is used to detect whether abandoned objects exist when a person target appears in the video stream information. In one embodiment, the abandoned object detection module 1130 can be used to perform the operation S230 described above, which will not be repeated here.

[0133] The video upload module 1140 is used to continuously upload the video stream from which the person appears in the video stream information to a remote server when a person target appears in the video stream information, so that the remote server can monitor the movement trajectory of the person target. In one embodiment, the video upload module 1140 can be used to perform the operation S240 described above, which will not be repeated here.

[0134] The motion trajectory acquisition module 1150 is used to acquire the motion trajectory from the remote server when the presence of the abandoned item is detected. In one embodiment, the motion trajectory acquisition module 1150 can be used to perform the operation S250 described above, which will not be repeated here.

[0135] In the embodiments of this disclosure, compared with the prior art, this disclosure includes at least the following advantages: 1. By monitoring the movement trajectory of personnel targets within the monitoring area before starting the detection of abandoned items, the computational cost of abandoned item detection can be reduced, and the presence of abandoned items can be identified more accurately. 2. By using time as the detection dimension, abandoned items of personnel targets are detected, assisting in improving the operational capabilities of the network. 3. The target tracking, which has a high computational cost, is placed on a remote server for identification, making up for the shortcomings of the limited computing power of local servers.

[0136] According to an embodiment of this disclosure, the personnel target determination module 1120 is further configured to acquire a reference image, wherein the reference image is an image in which the personnel target and / or the abandoned item does not exist in the monitoring area corresponding to the video stream information; calculate the difference between the reference image and the video frame image based on a frame difference algorithm to obtain a first difference; and determine that the personnel target appears in the video stream information when the first difference is outside a preset error range.

[0137] According to embodiments of this disclosure, the plurality of video frame images includes at least a first video frame image and a second video frame image, the first video frame image and the second video frame image are sequentially continuous, the first video frame image is the last video frame image in which the target customer is within the monitoring area, and the abandoned item detection module 1130 is further configured to determine whether the target person has left the monitoring area; if the target person has left the monitoring area, acquire the second video frame image; acquire the comparison image; calculate the difference between the comparison image and the second video frame image based on a frame difference algorithm to obtain a second difference; and if the second difference is outside a preset error range, determine that the abandoned item appears in the video stream information.

[0138] According to an embodiment of this disclosure, the personnel target judgment module is further configured to divide the video frame image and the control image into a preset number of sub-unit regions according to a preset division method, wherein the sub-unit region includes a unit identifier; and to calculate the difference between the sub-unit regions in the control image and the sub-unit regions in the video frame image one by one based on the frame difference algorithm according to the correspondence of the unit identifiers, so as to obtain a first sub-difference value.

[0139] According to an embodiment of this disclosure, the personnel target determination module 1120 is further configured to determine that the personnel target appears in the video stream information when the first sub-difference value is outside a preset error range, and to stop calculating the first sub-difference value of the remaining sub-unit regions.

[0140] According to an embodiment of this disclosure, the abandoned item detection module 1130 is further configured to divide the second video frame image and the control image into a preset number of sub-unit regions according to a preset division method, wherein the sub-unit region includes a unit identifier; and to calculate the difference between the sub-unit regions in the control image and the sub-unit regions in the video frame image one by one based on the frame difference algorithm according to the correspondence of the unit identifiers, thereby obtaining a second sub-difference value.

[0141] According to an embodiment of this disclosure, the abandoned item detection module 1130 is further configured to determine that the person target appears in the video stream information when the second sub-difference value is outside a preset error range, and to stop calculating the second sub-difference value of the remaining sub-unit area.

[0142] According to an embodiment of this disclosure, the device further includes: an alarm module, configured to detect whether there is a left-behind item when a person target appears in the video stream information, and further includes: issuing a left-behind item alarm command when the left-behind item is detected.

[0143] According to an embodiment of this disclosure, the device further includes: a leftover items classification module, used to identify the leftover items based on a preset leftover items recognition model and output a leftover items classification when the presence of the leftover items is detected.

[0144] According to embodiments of this disclosure, any multiple modules among the video receiving module 1110, personnel target judgment module 1120, abandoned item detection module 1130, video uploading module 1140, and motion trajectory acquisition module 1150 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the video receiving module 1110, personnel target judgment module 1120, abandoned item detection module 1130, video uploading module 1140, and motion trajectory acquisition module 1150 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the video receiving module 1110, the personnel target judgment module 1120, the abandoned item detection module 1130, the video uploading module 1140, and the motion trajectory acquisition module 1150 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0145] Figure 12 A block diagram schematically illustrates an electronic device suitable for implementing a method for detecting abandoned items according to an embodiment of the present disclosure.

[0146] like Figure 12 As shown, an electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage portion 1208 into a random access memory (RAM) 1203. The processor 1201 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1201 may also include onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0147] RAM 1203 stores various programs and data required for the operation of electronic device 1200. Processor 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Processor 1201 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1202 and / or RAM 1203. It should be noted that the programs may also be stored in one or more memories other than ROM 1202 and RAM 1203. Processor 1201 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0148] According to embodiments of this disclosure, the electronic device 1200 may further include an input / output (I / O) interface 1205, which is also connected to the bus 1204. The electronic device 1200 may also include one or more of the following components connected to the I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1210 as needed so that computer programs read from it can be installed into the storage section 1208 as needed.

[0149] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0150] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1202 and / or RAM 1203 and / or one or more memories other than ROM 1202 and RAM 1203 described above.

[0151] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0152] When the computer program is executed by the processor 1201, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0153] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1209, and / or installed from the removable medium 1211. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0154] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1209, and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0155] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0158] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for detecting abandoned items, the method being applied to a network server, the method comprising: Receive video stream information; The frame difference algorithm is used to determine whether a human target appears in the video stream information; When a person is present in the video stream, the system detects whether any items are left behind. This detection includes: a remote server determining whether the person has left; and if the person has left, the branch server using a frame difference algorithm to detect whether any items are left behind. When a person is detected in the video stream, the video stream from which the person is detected is continuously uploaded to a remote server, whereby the remote server monitors the movement trajectory of the person using a trajectory tracking algorithm; wherein the computational cost of the trajectory tracking algorithm is greater than that of the frame difference algorithm; and If the presence of the abandoned item is detected, the movement trajectory is obtained from the remote server.

2. The method according to claim 1, wherein, The video stream information includes multiple video frame images. The step of determining whether a person target appears in the video stream information includes: Obtain a comparison image, which is an image of the person target and / or the abandoned item that does not exist in the monitoring area corresponding to the video stream information; The difference between the reference image and the video frame image is calculated based on the frame difference algorithm to obtain a first difference; and If the first difference is outside the preset error range, it is determined that the person target appears in the video stream information.

3. The method according to claim 2, wherein, The plurality of video frame images includes at least a first video frame image and a second video frame image, the first video frame image and the second video frame image are sequentially consecutive, and the first video frame image is the last video frame image in which the person target is within the monitoring area. The detection of whether any items were left behind includes: Determine whether the person or target has left the monitored area; When the person leaves the monitored area, the second video frame image is acquired; Obtain the comparison image; The difference between the comparison image and the second video frame image is calculated based on the frame difference algorithm to obtain the second difference; and If the second difference is outside the preset error range, it is determined that the abandoned item appears in the video stream information.

4. The method according to claim 2, wherein, The step of calculating the difference between the reference image and the video frame image based on the frame difference algorithm to obtain the first difference includes: According to a preset division method, the video frame image and the comparison image are respectively divided into a preset number of sub-unit regions, wherein each sub-unit region includes a unit identifier; and According to the correspondence of the unit identifiers, the difference between the sub-unit regions in the comparison image and the sub-unit regions in the video frame image is calculated one by one based on the frame difference algorithm to obtain the first sub-difference value.

5. The method according to claim 4, wherein, The step of determining that the person target appears in the video stream information when the first difference is outside a preset error range includes: If the first sub-difference value is outside the preset error range, it is determined that the person target appears in the video stream information, and the calculation of the first sub-difference value of the remaining sub-unit regions is stopped.

6. The method according to claim 3, wherein, The step of calculating the difference between the comparison image and the second video frame image based on the frame difference algorithm to obtain the second difference includes: According to a preset division method, the second video frame image and the comparison image are respectively divided into a preset number of sub-unit regions, wherein each sub-unit region includes a unit identifier; and According to the correspondence of the unit identifiers, the difference between the sub-unit regions in the comparison image and the sub-unit regions in the video frame image is calculated one by one based on the frame difference algorithm to obtain the second sub-difference value.

7. The method according to claim 6, wherein, The step of determining that the abandoned item appears in the video stream information when the second difference is outside the preset error range includes: If the second sub-difference value is outside the preset error range, it is determined that the person target appears in the video stream information, and the calculation of the second sub-difference value of the remaining sub-unit regions is stopped.

8. The method according to any one of claims 1 to 7, wherein, When a person is detected in the video stream, after detecting whether any items have been left behind, the process further includes: If abandoned items are detected, an alarm command for abandoned items will be issued.

9. The method according to any one of claims 1 to 7, wherein, If a person is detected in the video stream, after detecting whether any items have been left behind, the process further includes: If the abandoned items are detected, the abandoned items are identified based on a preset abandoned item recognition model, and the abandoned item classification is output.

10. The method according to claim 9, wherein, The training method for the preset abandoned item recognition model includes: Obtain training samples and sample labels, wherein the training samples include images of historical artifacts, and the sample labels include artifact categories; and The images of the historical artifacts and the categories of the artifacts are used as input data to train the model until a preset number of training iterations are reached.

11. A device for detecting abandoned items, the device being installed in a network server, the device comprising: The video receiving module is used to receive video stream information; The personnel target determination module is used to determine whether a personnel target appears in the video stream information based on the frame difference algorithm; A module for detecting abandoned items is used to detect whether abandoned items exist when a person is present in the video stream information; wherein, the detection of whether abandoned items exist includes: a remote server determining whether the person has left; and if the person has left, the branch server detecting whether abandoned items exist using a frame difference algorithm; A video uploading module is used to continuously upload the video stream from which the person appears in the video stream to a remote server when a person target is present, so that the remote server can monitor the movement trajectory of the person target using a trajectory tracking algorithm; wherein the computational cost of the trajectory tracking algorithm is greater than that of the frame difference algorithm; and The motion trajectory acquisition module is used to acquire the motion trajectory from the remote server when the presence of the abandoned item is detected.

12. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images