Method for detecting object retention

By using the partial completion network and the content completion network to segment and count instances of trapped objects, the missed detection problem caused by object occlusion is solved, the missed detection rate is reduced, and the accuracy of object retention detection is improved.

CN114663906BActive Publication Date: 2024-12-03CHINA TELECOM CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011527024.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-22
Publication Date
2024-12-03
Estimated Expiration
2040-12-22

AI Technical Summary

Technical Problem

In the prior art, the leakage detection rate of object retention detection is high, especially when objects are blocked and overlapped, which leads to failure of the target tracking algorithm, resulting in high false alarm rate or high leakage detection rate.

Method used

The partial completion network PCNet-M and content completion network PCNet-C are used for instance segmentation and RGB image completion, and the trapped objects in the camera screen are processed, and the trapped objects are counted through semantic segmentation and target classification, and the entry and exit cameras are compared. The number of objects in the screen is determined by whether there are objects stuck.

Benefits of technology

It effectively reduces the missed detection rate of object retention detection, can correctly count the retention objects under occlusion phenomenon, and improves the accuracy of detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114663906B_ABST
    Figure CN114663906B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a method for detecting object retention, comprising: determining the number of retained objects in a camera picture before a person enters a frame; determining the number of retained objects in the camera picture after the person exits the frame; if the number of retained objects in the camera picture before the person enters the frame is less than the number of retained objects in the camera picture after the person exits the frame, it is presumed that the person has retained an object; wherein, when determining the number of retained objects in the camera picture, if there is occlusion between the retained objects, then: using a partial completion network PCNet‑M to obtain an instance segmentation image mask for a camera picture image, using a partial completion network PCNet‑M to obtain an occlusion relationship sequence diagram of all retained objects for the camera picture image, using a content completion network PCNet‑C to perform RGB image completion on the retained objects to obtain decomposed objects, performing target classification on the decomposed objects, and counting the objects classified by the target.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates generally to the field of video processing, and more particularly to a method for detecting object retention. Background Art

[0002] With the rapid development of modern science and technology, and the gradual improvement and refinement of image and video acquisition technologies, smart security video surveillance has become widely used in people's daily lives. However, behaviors such as littering and illegal road use are common, impacting urban public health, increasing the difficulty of public safety management, threatening the lives and property of citizens, and even causing irreparable damage in some critical areas.

[0003] At present, there are two main methods for determining whether objects are stranded, such as littering and vehicles illegally occupying the road.

[0004] The first is to rely on regular manual inspections to solve problems like lost objects and road obstructions. However, this method is not only time-consuming and labor-intensive, but also requires high labor costs. Furthermore, manual inspections can only be performed periodically, which can easily lead to omissions or negligence.

[0005] The second approach uses a deep learning-based object detection algorithm in conjunction with target tracking. The problem with this approach is that when occlusion or overlap of an object causes the tracking algorithm to lose track and regenerate a unique ID, the object retention behavior of the grasper fails, resulting in high false alarms and missed detections. In this technical field, the false alarm rate refers to the proportion of false alarms in the detection results, while the missed detection rate refers to the proportion of all object retention events that the algorithm model missed. Summary of the Invention

[0006] A brief overview of the present disclosure is provided below to provide a basic understanding of some aspects of the present disclosure. However, it should be understood that this overview is not an exhaustive overview of the present disclosure. It is not intended to identify key or important parts of the present disclosure, nor is it intended to limit the scope of the present disclosure. Its purpose is simply to present certain concepts of the present disclosure in a simplified form as a prelude to the more detailed description that will be given later.

[0007] One of the purposes of the present disclosure is to solve one of the above-mentioned problems in the prior art, such as the high missed detection rate.

[0008] According to one aspect of the present disclosure, a method for detecting object retention is provided, comprising the steps of:

[0009] Determine the number of objects remaining in the camera's view before a person enters the camera's view;

[0010] Determine the number of objects remaining in the camera view after a person leaves the camera view;

[0011] If the number of objects retained in the camera frame before the person enters the camera frame is less than the number of objects retained in the camera frame after the person leaves the camera frame, it is presumed that the person has retained an object;

[0012] Among them, when determining the number of stranded objects in the camera image, if there is occlusion between the stranded objects, then: the partial completion network PCNet-M is used to obtain the instance segmentation image mask for the camera image, the partial completion network PCNet-M is used to obtain the occlusion relationship sequence diagram of all stranded objects for the camera image, the content completion network PCNet-C is used to complete the RGB image of the stranded objects to obtain the decomposed independent and complete objects, the decomposed independent and complete objects are classified, and the classified objects are counted.

[0013] According to another aspect of the present disclosure, there is provided an apparatus for detecting a retained object, comprising:

[0014] a memory having instructions stored thereon; and

[0015] The processor is configured to execute instructions stored in the memory to perform the above method for detecting object retention.

[0016] According to another aspect of the present disclosure, a computer-readable storage medium is provided, comprising computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors are caused to perform the above-mentioned method for detecting object retention.

[0017] According to another aspect of the present disclosure, a computer program product is provided, comprising computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform the above-mentioned method for detecting object retention.

[0018] According to the above solution of the present disclosure, the missed detection rate of object retention detection can be reduced.

[0019] Other features and advantages of the present invention will become apparent from the following detailed description of preferred embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0022] Figure 1 The process flow of the method for detecting object retention is shown;

[0023] Figure 2 The processing flow when there is occlusion between the retained objects is shown;

[0024] Figure 3 An exemplary configuration of a computing device capable of implementing an embodiment according to the present disclosure is shown. DETAILED DESCRIPTION

[0025] The following detailed description is made with reference to the accompanying drawings, and the following detailed description is provided to assist in a comprehensive understanding of the various exemplary embodiments of the present disclosure. The following description includes various details to assist in understanding, but these details are considered to be examples only and are not intended to limit the present disclosure, which is defined by the appended claims and their equivalents. The words and phrases used in the following description are intended only to enable a clear and consistent understanding of the present disclosure. In addition, descriptions of well-known structures, functions, and configurations may be omitted for clarity and brevity. Those skilled in the art will recognize that various changes and modifications may be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0026] Figure 1 The process flow of the method for detecting object retention is shown.

[0027] When a person passes through the camera's field of view, the person will enter the camera screen in the person-entering frame state, the person is in the camera screen in the person-in-frame state, and the person leaves the camera screen in the person-out-frame state.

[0028] The video data from the camera can record the entire process including the person entering the camera screen, the person being in the camera screen, and the person leaving the camera screen.

[0029] The present disclosure processes video data from a camera to detect whether a person entering the camera's field of view has engaged in an act of retaining an object, such as garbage or a vehicle.

[0030] like Figure 1 As shown in step S101 , the present disclosure determines the number of objects remaining in the camera image before a person enters the camera image.

[0031] Specifically, for example, by applying the image semantic segmentation model BiseNet and smoothing processing, we can infer people passing through the camera image and define the "in-frame" state for people entering the camera image, the "in-frame" state for people remaining in the camera image, and the "out-frame" state for people leaving the camera image. In this field, image semantic segmentation refers to segmenting various objects in an image using different colors. For those skilled in the art, the image semantic segmentation model BiseNet is a well-known image semantic segmentation model in the art.

[0032] Based on the results of image semantic segmentation, the target tracking model can assign an identification ID to the detected object target, and can, for example, associate the object target through the identification ID to achieve the effect of counting single objects.

[0033] In the case of occlusion between retained objects, the present disclosure uses the partial completion network PCNet-M to obtain instance segmentation image masks for the camera screen image, and can infer the number of retained objects in the camera screen.

[0034] Furthermore, the present disclosure can further preferably obtain an occlusion relationship sequence diagram of all retained objects based on the camera image using the partial completion network PCNet-M, and use the content completion network PCNet-C to complete the RGB image of the retained objects to obtain independent and complete decomposed objects. For those skilled in the art, the partial completion network PCNet-M and the content completion network PCNet-C are both known neural networks.

[0035] In addition, it is further preferred that the present disclosure can also count objects that have been classified as targets. Target classification is a target recognition algorithm based on deep learning.

[0036] It can be seen that even when there is occlusion between objects, the present disclosure can still perform counting correctly, thereby solving the problem of missed detection caused by occlusion between objects.

[0037] like Figure 1 As shown in step S102 in the embodiment, the number of objects remaining in the camera image after the person leaves the camera image is further determined.

[0038] The way in which the present disclosure determines the number of objects stranded in the camera screen in step S102 is similar to that in step S101. In particular, in the case where there is occlusion between the stranded objects, the present disclosure uses the partial completion network PCNet-M to obtain an instance segmentation image mask for the camera screen image, and infers the number of objects stranded in the camera screen. Similarly, the present disclosure can also further obtain an occlusion relationship sequence diagram of all stranded objects for the camera screen image based on the partial completion network PCNet-M. The present disclosure can also use the content completion network PCNet-C to complete the RGB image of the stranded objects, so as to decompose the camera screen scene of the specified area into multiple independent and complete objects, obtain the decomposed independent and complete objects, and obtain an occlusion sorting diagram. The present disclosure can also perform target classification on the decomposed objects, and the present disclosure can also count the objects that have been classified by the target.

[0039] The present disclosure can also determine whether there is a person or object retention behavior by determining whether the number of objects in the designated area increases. Specifically, Figure 1 As shown in step S103 of the present disclosure, the present disclosure compares the number of objects retained in the camera image before the person enters the camera image with the number of objects retained in the camera image after the person leaves the camera image. If the number of objects retained in the camera image before the person enters the camera image is less than the number of objects retained in the camera image after the person leaves the camera image, it means that the number of objects retained in the camera image has increased, and it is thus inferred that the person has retained objects (S104). If the number of objects retained in the camera image before the person enters the camera image is not less than the number of objects retained in the camera image after the person leaves the camera image, it means that the number of objects retained in the camera image has not increased, and it is thus inferred that the person has not retained objects (S105).

[0040] For example, if no objects were present before a person entered the frame, but objects remain after the person exits the frame, then it can be inferred that the person engaged in object retention. For another example, if objects were present before the person entered the frame, and objects remain present after the person exits the frame, but there is no occlusion, the person's object retention behavior can be determined by determining whether the number of objects remaining in the specified area increases. For another example, if objects were present before the person entered the frame, and objects remain present after the person exits the frame, and there is occlusion, then the partial completion network PCNet-M can be used to obtain instance segmentation image masks for the camera image and infer the number of objects remaining in the camera image. The partial completion network PCNet-M can also be used to further obtain an occlusion relationship sequence graph for all objects remaining in the camera image. The content completion network PCNet-C can also be used to complete the RGB images of the objects, decomposing the camera scene in the specified area into multiple independent and complete objects, obtaining the decomposed independent and complete objects. The decomposed objects can also be further classified and counted.

[0041] As an optional step, when it is presumed that a person has retained an object, the present disclosure finds the frame in which the increase in the number of retained objects is detected for the first time after the person enters the camera screen as a key frame, and outputs video data including the key frame and a predetermined range before and after the key frame. For example, the position of the key frame can be obtained by traversing the image sequence and judging a certain feature mark. The predetermined range before the key frame and the predetermined range after the key frame can be, for example, a video range of 1 second to 10 seconds, such as a video range of 3 seconds, 5 seconds or 7 seconds. The length of the predetermined range before the key frame may be equal to or different from the length of the predetermined range after the key frame. Moreover, the length of the predetermined range before the key frame and the length of the predetermined range after the key frame can both be set by the user.

[0042] For example, if a person is suspected of retaining an object, the present disclosure can issue an alert and use the aforementioned video data, including the key frame and a predetermined range before and after the key frame, as video evidence corresponding to the alert. This video evidence fully records the entire process of the person retaining the object.

[0043] As an optional step, when it is presumed that a person has retained an object, the present disclosure can also find the picture frame in which the retained object is first detected after the person enters the camera image, and further use the picture frame as evidence corresponding to the alarm information. Such a picture frame can further prove where the retained object first appeared after entering the camera image. For example, if the retained object is a bag of garbage, the bag of garbage may be held in the hand of the person when it is first detected. As an example, the position of a specified picture frame can be obtained by traversing the picture sequence and judging a certain feature marker.

[0044] Instructions attached Figure 2 A preferred processing flow when there is an occlusion relationship between the remaining objects is further specifically illustrated.

[0045] In a preferred example of the present invention, a self-supervised completion mechanism is adopted.

[0046] Specifically, if Figure 2 As shown in step S201, the instance segmentation image mask of the camera screen image can be output based on the partial completion network PCNet-M, and all the retained objects in the camera screen can be inferred.

[0047] Furthermore, it is further preferred that Figure 2 In step S202, the occlusion relationship of all retained objects in the camera image is inferred based on the partial completion network PCNet-M.

[0048] On this basis, if Figure 2 As shown in step S203, the content completion network PCNet-C can be further used to complete the RGB image of the retained object. Figure 2 As shown in step S204, the scene can be decomposed into multiple independent and complete objects, thereby obtaining the decomposed independent and complete objects. The present disclosure can also perform target classification on the decomposed independent and complete objects, and count the target-classified objects.

[0049] Furthermore, based on the occlusion relationships of all the objects retained in the camera image inferred in step S202 , an occlusion relationship sequence diagram can be obtained.

[0050] The above specifically illustrates an intelligent object retention detection method according to a preferred embodiment of the present invention, which adopts semantic segmentation, key frame backtracking, target classification and self-supervised completion mechanism to detect the behavior of people and objects being retained, and can reduce the missed detection rate caused by occlusion between objects.

[0051] Figure 3 An exemplary configuration of a computing device 300 capable of implementing embodiments according to the present disclosure is shown.

[0052] Computing device 300 is an example of a hardware device to which the above-described aspects of the present disclosure can be applied. Computing device 300 can be any machine configured to perform processing and / or computation. Computing device 300 can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal data assistant (PDA), a smartphone, an in-vehicle computer, or a combination thereof.

[0053] like Figure 3 As shown, the computing device 300 may include one or more components that can be connected or communicated with the bus 302 via one or more interfaces. The bus 302 may include, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus. The computing device 300 may include, for example, one or more processors 304, one or more input devices 306, and one or more output devices 308. The one or more processors 304 may be any type of processor and may include, but is not limited to, one or more general-purpose processors or special-purpose processors (such as a dedicated processing chip). The processor 304 may be configured to perform, for example Figure 2 or Figure 3 The method shown. Input device 306 can be any type of input device capable of inputting information into a computing device, and can include, but is not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote controller. Output device 308 can be any type of device capable of presenting information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer.

[0054] The computing device 300 may also include or be connected to a non-transitory storage device 314, which may be any non-transitory storage device that can implement data storage and may include, but is not limited to, a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a magnetic tape or any other magnetic medium, a compact disk or any other optical medium, a cache memory and / or any other storage chip or module, and / or any other medium from which a computer can read data, instructions and / or code. The computing device 300 may also include a random access memory (RAM) 310 and a read-only memory (ROM) 312. The ROM 312 may store programs, utilities or processes to be executed in a non-volatile manner. The RAM 310 may provide volatile data storage and store instructions related to the operation of the computing device 300. The computing device 300 may also include a network / bus interface 316 coupled to a data link 318. The network / bus interface 316 may be any type of device or system capable of enabling communication with external devices and / or networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication device, and / or a chipset (such as a Bluetooth TM equipment, 802.11 equipment, WiFi equipment, WiMax equipment, cellular communication facilities, etc.).

[0055] The present disclosure may be implemented as any combination of an apparatus, a system, an integrated circuit, and a computer program on a non-transitory computer-readable medium. One or more processors may be implemented as an integrated circuit (IC), an application-specific integrated circuit (ASIC), or a large-scale integrated circuit (LSI), a system LSI, a super LSI, or a ultra-LSI component that performs some or all of the functions described in the present disclosure.

[0056] The present disclosure includes the use of software, applications, computer programs, or algorithms. The software, applications, computer programs, or algorithms can be stored on a non-transitory computer-readable medium to enable a computer, such as one or more processors, to perform the steps described above and in the accompanying figures. For example, one or more memories can store the software or algorithm in the form of executable instructions, and one or more processors can execute a set of instructions for the software or algorithm to provide various functions according to the embodiments described in the present disclosure.

[0057] Software and computer programs (which may also be referred to as programs, software applications, applications, components, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural language, an object-oriented programming language, a functional programming language, a logic programming language, or assembly language or machine language. The term "computer-readable medium" refers to any computer program product, apparatus, or device for providing machine instructions or data to a programmable data processor, such as magnetic disks, optical disks, solid-state storage devices, memories, and programmable logic devices (PLDs), including computer-readable media that receive machine instructions as computer-readable signals.

[0058] For example, computer-readable media may include dynamic random access memory (DRAM), random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store the desired computer-readable program code in the form of instructions or data structures and can be accessed by a general-purpose or special-purpose computer or a general-purpose or special-purpose processor. As used herein, disk or disc includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks generally reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above are also included within the scope of computer-readable media.

[0059] The subject matter of the present disclosure is provided as an example of devices, systems, methods, and programs for performing the features described in the present disclosure. However, in addition to the features described above, other features or variations are also contemplated. It is contemplated that any emerging technology may be used to implement the components and functions of the present disclosure in place of any of the technologies described above.

[0060] In addition, the above description provides examples and does not limit the scope, applicability or configuration set forth in the claims. Without departing from the spirit and scope of this disclosure, the function and arrangement of the elements discussed can be changed. Various embodiments can appropriately omit, replace or add various processes or components. For example, features described with respect to certain embodiments can be combined in other embodiments.

[0061] Similarly, while operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed to achieve desired results. In certain circumstances, multitasking and parallel processing can be advantageous.

Claims

1. A method for detecting object retention, comprising the steps of: Determine the number of objects that remain in the camera view before a person enters the camera view; Determine the number of objects remaining in the camera view after a person leaves the camera view; If the number of objects retained in the camera screen before the person enters the camera screen is less than the number of objects retained in the camera screen after the person leaves the camera screen, it is presumed that the person has retained objects; Among them, when determining the number of stranded objects in the camera screen, if there is occlusion between the stranded objects, then: use the partial completion network PCNet-M to obtain the instance segmentation image mask for the camera screen image, use the partial completion network PCNet-M to obtain the occlusion relationship sequence diagram of all stranded objects for the camera screen image, use the content completion network PCNet-C to complete the RGB image of the stranded objects to obtain the decomposed independent and complete objects, classify the decomposed independent and complete objects, and count the objects classified by the target.

2. The method according to claim 1, further comprising: If it is inferred that the person has retained an object, the frame in which the increase in the number of retained objects is detected for the first time after the person enters the camera screen is found as a key frame, and video data including the key frame and a predetermined range before and after the key frame is output.

3. The method according to claim 2, further comprising: An alarm message is issued, and the video data is used as video evidence corresponding to the alarm message.

4. The method according to claim 1, further comprising: If it is inferred that the person has retained an object, then find the picture frame in which the retained object is first detected after the person enters the camera screen.

5. The method according to claim 4, further comprising: An alarm message is issued, and the picture frame is used as evidence corresponding to the alarm message.

6. The method according to claim 1, wherein: The video frame of the video data from the camera indicates a person-entering-frame state where a person enters the camera screen, a person-in-frame state where a person is in the camera screen, or a person-out-of-frame state where a person leaves the camera screen.

7. A device for detecting object retention, comprising: a memory having instructions stored therein; as well as A processor configured to execute instructions stored on the memory to perform the method according to any one of claims 1 to 6.

8. A computer-readable storage medium comprising computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 6.

9. A computer program product comprising computer executable instructions, which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 6.