Video analysis method and system

Through video analysis methods, the image collected by the camera is detected and tracked, and combined with image semantic segmentation and multi-camera data, the problem of insufficient accuracy in behavior recognition in complex dynamic scenarios is solved, and production safety and judgment reliability are improved.

CN118968394AInactive Publication Date: 2024-11-15HITACHI LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310547173.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-15
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art is difficult to accurately identify the target object's violations in complex dynamic scenarios, resulting in insufficient accuracy of whether the behavior is violated.

Method used

Through the video analysis method, the images collected by the camera are obtained for feature detection, the area elements, subject elements and judgment elements are identified, the subject elements are tracked into the work area, and the job is determined based on the judgment elements in the multi-frame image. The image semantic segmentation is used Technology and data collected by multiple cameras improve judgment accuracy.

Benefits of technology

Compliance operation detection in complex dynamic scenarios is realized, production safety is improved, and the reliability of judgment results is improved through the combination of space-time logic judgment and multi-camera data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968394A_ABST
    Figure CN118968394A_ABST
Patent Text Reader

Abstract

The invention provides a video analysis method and system. The method comprises the following steps: acquiring an image acquired by a camera; element detection is carried out on the image, elements existing in each frame of image are obtained, and the elements comprise area elements, main body elements and judgment elements; the area elements at least comprise an operation area, and the operation area is an area needing compliance operation detection; the main body element is a target object; the judgment element is an element for judging whether the operation is compliant or not, and the judgment element comprises a compliant state and a violation state; and under the condition that the operation area and the main body element are detected, tracking the main body element, and after the main body element enters the operation area, determining whether the operation of the main body element is compliant or not according to the judgment element in the multi-frame image. According to the invention, compliance operation detection in a complex dynamic scene can be realized, and the safety of production operation is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video analysis method and system. Background Art

[0002] Behavior recognition of the target object helps assist monitoring. For example, in factories and warehouses, it is usually necessary to monitor whether workers have violated regulations during specific working hours and work areas to reduce production safety risks.

[0003] In the prior art, a Chinese patent application with publication number CN115019400A discloses a method and device for detecting illegal behavior, which specifically discloses the following scheme: capturing a target object; obtaining a first image and multiple second images of the target object; based on the first image, identifying the environment in which the target object is located; based on the environment, obtaining a detection model corresponding to the environmental parameters; based on the multiple second images, identifying the behavior of the target object to obtain the posture features of the target object; inputting the posture features into the detection model for detection to determine whether the behavior of the target object is illegal. This application aims to solve the technical problem of insufficient accuracy in determining whether a behavior is illegal in the prior art.

[0004] The above solution mainly recognizes the posture features of the target object through static images, and then determines whether there is any violation, which is difficult to apply to complex dynamic scenes. Summary of the invention

[0005] At least one embodiment of the present application provides a video analysis method and system for implementing compliance operation detection in complex dynamic scenarios and improving production safety.

[0006] According to one aspect of the present application, at least one embodiment provides a video analysis method, including:

[0007] Get the image captured by the camera;

[0008] Performing element detection on the image to obtain elements existing in each frame of the image, wherein the elements include regional elements, main elements and determination elements; the regional elements at least include an operation area, and the operation area is an area that needs to be detected for compliance operations; the main elements are target objects; the determination elements are elements for determining whether the operation is compliant, and the determination elements include compliance status and violation status;

[0009] When the operation area and the main element are detected, the main element is tracked, and after the main element enters the operation area, it is determined whether the operation of the main element is compliant based on the determination elements in the multiple frames of images.

[0010] Optionally, the area element further includes a preparation area, which is an area for entering the working area from outside the working area;

[0011] Performing element detection on the image includes:

[0012] When both the working area and the preparation area are pre-set fixed areas, determining the working area and the preparation area in the image according to the pre-set fixed areas;

[0013] When both the working area and the preparation area are non-fixed areas, image segmentation is performed on regional elements using image semantic segmentation technology to determine the working area and the preparation area in the image.

[0014] Optionally, the above method further includes:

[0015] According to the tracking trajectory obtained by tracking the main element, when the main element enters the operation area through the preparation area and the time length in the operation area is greater than the preset time length threshold, the step of determining whether the operation of the main element is compliant based on the judgment elements in the multiple frames of images is executed.

[0016] Optionally, performing element detection on the image further includes:

[0017] The target object recognition technology is used to detect the main elements of the image to determine whether the main elements exist in the image. When the main elements exist in the image, the target objects corresponding to the determination elements are further detected in the image through the object recognition technology to obtain the determination elements corresponding to the image.

[0018] Optionally, determining whether the operation of the subject element is compliant according to the determination elements in the multiple frames of images includes:

[0019] For each determination factor, a queue consisting of determination factors corresponding to each frame of the multiple images is obtained; and whether the determination factor meets the operation requirements is determined according to whether the proportion of compliance status determination factors in the queue is greater than a preset first threshold value;

[0020] When any of the determination elements does not meet the operation requirements, it is determined that the operation of the main element is non-compliant;

[0021] When all the determination factors meet the operation requirements, it is determined that the operation of the main elements is compliant.

[0022] Optionally, there are multiple cameras;

[0023] Determine whether the operation of the subject element is compliant based on the judgment elements in the multiple frames of images, including:

[0024] For each determination factor, determine whether the determination factor meets the job requirements according to the following steps: for each camera, obtain a queue consisting of determination factors corresponding to each frame of multiple frames of images captured by the camera; determine a determination result corresponding to the camera according to whether the proportion of compliance status determination factors in the queue is greater than a preset first threshold, and the determination result is whether the job requirements are met or not met; determine whether the determination factor meets the job requirements according to a determination result that accounts for a larger proportion of the determination results corresponding to the multiple cameras;

[0025] When any of the determination elements does not meet the operation requirements, it is determined that the operation of the main element is non-compliant;

[0026] When all the determination factors meet the operation requirements, it is determined that the operation of the main elements is compliant.

[0027] Optionally, when detecting the target object in the image by using an object recognition technology, further obtaining the confidence of the determination factor corresponding to the image;

[0028] If, among the judgment results corresponding to the multiple cameras, the proportion of the judgment results that meet the operation requirements is the same as the proportion of the judgment results that do not meet the operation requirements, the method further includes:

[0029] For a camera whose judgment result is that the operation requirements are met, the average confidence of the compliance status in the judgment factor corresponding to the multiple frames of images collected by the camera is calculated to obtain a first average confidence corresponding to the camera; the first average confidence corresponding to all cameras whose judgment results are that the operation requirements are met is summed to obtain a first score;

[0030] For a camera whose judgment result is that the operation requirements are not met, the average confidence of the violation status in the judgment factor corresponding to the multiple frames of images collected by the camera is calculated to obtain a second average confidence corresponding to the camera; the second average confidence corresponding to all cameras whose judgment results are that the operation requirements are not met is summed to obtain a second score;

[0031] When the first score is greater than the second score, determining that the determination factor meets the operation requirement;

[0032] When the first score is not greater than the second score, it is determined that the determination factor does not meet the operation requirement.

[0033] Optionally, the above method further includes:

[0034] When any of the determination elements does not meet the operation requirements, an alarm is generated that any of the elements does not meet the operation requirements and / or the operation of the main element is not compliant.

[0035] According to another aspect of the present application, at least one embodiment provides a video analysis system, including:

[0036] An image acquisition module is used to acquire images captured by a camera;

[0037] An element detection module is used to perform element detection on the image to obtain elements existing in each frame of the image, wherein the elements include regional elements, main elements and determination elements; the regional elements at least include an operation area, and the operation area is an area that needs to be detected for compliance operations; the main elements are target objects; the determination elements are elements for determining whether the operation is compliant, and the determination elements include compliance status and violation status;

[0038] The compliance judgment module is used to track the main element when the operation area and the main element are detected, and after the main element enters the operation area, determine whether the operation of the main element is compliant based on the judgment elements in multiple frames of images.

[0039] Optionally, the area element further includes a preparation area, which is an area from outside the working area to the working area; the element detection module is further used to:

[0040] When both the working area and the preparation area are pre-set fixed areas, determining the working area and the preparation area in the image according to the pre-set fixed areas;

[0041] When both the working area and the preparation area are non-fixed areas, image segmentation is performed on regional elements using image semantic segmentation technology to determine the working area and the preparation area in the image.

[0042] Optionally, in the above system, the compliance judgment module is also used to execute the step of determining whether the operation of the main element is compliant based on the judgment elements in the multiple frames of images when the main element enters the operating area through the preparation area and the time length in the operating area is greater than a preset time length threshold based on the tracking trajectory obtained by tracking the main element.

[0043] Optionally, the element detection module is also used to detect the main elements of the image through target object recognition technology to determine whether the main elements exist in the image, and when the main elements exist in the image, further detect the target objects in the image through object recognition technology based on the target objects corresponding to the determination elements to obtain the determination elements corresponding to the image.

[0044] Optionally, the compliance judgment module is further used to:

[0045] For each determination factor, a queue consisting of determination factors corresponding to each frame of the multiple images is obtained; and whether the determination factor meets the operation requirements is determined according to whether the proportion of compliance status determination factors in the queue is greater than a preset first threshold value;

[0046] When any of the determination elements does not meet the operation requirements, it is determined that the operation of the main element is non-compliant;

[0047] When all the determination factors meet the operation requirements, it is determined that the operation of the main elements is compliant.

[0048] Optionally, there are multiple cameras;

[0049] The compliance judgment module is further used to:

[0050] For each determination factor, determine whether the determination factor meets the job requirements according to the following steps: for each camera, obtain a queue consisting of determination factors corresponding to each frame of multiple frames of images captured by the camera; determine a determination result corresponding to the camera according to whether the proportion of compliance status determination factors in the queue is greater than a preset first threshold, and the determination result is whether the job requirements are met or not met; determine whether the determination factor meets the job requirements according to a determination result that accounts for a larger proportion of the determination results corresponding to the multiple cameras;

[0051] When any of the determination elements does not meet the operation requirements, it is determined that the operation of the main element is non-compliant;

[0052] When all the determination factors meet the operation requirements, it is determined that the operation of the main elements is compliant.

[0053] Optionally, the element detection module is further used to further obtain the confidence of the determination element corresponding to the image when detecting the target object in the image through the object recognition technology;

[0054] The compliance judgment module is further configured to: when, among the judgment results corresponding to the multiple cameras, the proportion of the judgment results that meet the operation requirements is the same as the proportion of the judgment results that do not meet the operation requirements:

[0055] For a camera whose judgment result is that the operation requirements are met, the average confidence of the compliance status in the judgment factor corresponding to the multiple frames of images collected by the camera is calculated to obtain a first average confidence corresponding to the camera; the first average confidence corresponding to all cameras whose judgment results are that the operation requirements are met is summed to obtain a first score;

[0056] For a camera whose judgment result is that the operation requirements are not met, the average confidence of the violation status in the judgment factor corresponding to the multiple frames of images collected by the camera is calculated to obtain a second average confidence corresponding to the camera; the second average confidence corresponding to all cameras whose judgment results are that the operation requirements are not met is summed to obtain a second score;

[0057] When the first score is greater than the second score, determining that the determination factor meets the operation requirement;

[0058] When the first score is not greater than the second score, it is determined that the determination factor does not meet the operation requirement.

[0059] Optionally, the above system further includes:

[0060] The alarm module is used to generate an alarm that the subject element is not wearing any target item and / or the subject element is not operating in compliance with regulations when the subject element is not wearing any target item.

[0061] According to another aspect of the present application, at least one embodiment provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the steps of the method described above are implemented.

[0062] Compared with the prior art, the video analysis method and system provided in the embodiment of the present application can realize compliance operation detection in complex dynamic scenes and improve the safety of production operations. In addition, the embodiment of the present application can improve the reliability of the judgment result by judging violations based on the spatiotemporal logic judgment method and the images collected by multiple cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0064] Figure 1 This is an example diagram of the arrangement of regions and cameras in an embodiment of the present application;

[0065] Figure 2 A schematic diagram of a flow chart of a video analysis method according to an embodiment of the present application;

[0066] Figure 3-4 An exemplary diagram of a fixed working area and a preparation area according to an embodiment of the present application;

[0067] Figure 5-6 An exemplary diagram of a non-fixed working area and a preparation area according to an embodiment of the present application;

[0068] Figure 7 An exemplary flow chart of a video analysis method according to an embodiment of the present application;

[0069] Figure 8 An example diagram of the movement trajectory of the main elements of the embodiment of the present application;

[0070] Fig. 9 This is a flow chart of a violation judgment method based on spatiotemporal logic according to an embodiment of the present application;

[0071] Fig.10 This is an example diagram where the main element appears in the working area without passing through the preparation area;

[0072] Fig.11 An example diagram determined to be compliant in an embodiment of the present application;

[0073] Fig.12 An example diagram of a violation determined in an embodiment of the present application;

[0074] Fig.13 Another example diagram of a violation determined in accordance with an embodiment of the present application;

[0075] Fig.14 A schematic diagram of the structure of a video analysis system according to an embodiment of the present application;

[0076] Fig.15 Another structural schematic diagram of the video analysis system according to an embodiment of the present application. DETAILED DESCRIPTION

[0077] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0078] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. "And / or" in the specification and claims represents at least one of the connected objects.

[0079] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0080] The following description provides examples and does not limit the scope, applicability, or configuration set forth in the claims. Changes may be made to the functions and arrangements of the elements discussed without departing from the spirit and scope of the present disclosure. Various examples may appropriately omit, replace, or add various procedures or components. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0081] The embodiment of the present application provides a video analysis method, which can realize compliance operation detection in complex dynamic scenes based on video data collected by one or more cameras, thereby improving production safety.

[0082] In the embodiment of the present application, one or more cameras may be provided to cover the possible working range of the target object (e.g., a worker) and collect video data of the area, wherein the video data generally includes multiple frames of images. When multiple cameras are used, the shooting angles of the cameras may be different. Figure 1An example of area and camera arrangement is shown, wherein a vehicle is parked in the possible working range 102 of the target object 106, and the vehicle includes a front area 101 and a roof area 103. The roof area 103 is the working area, and the working area is an area where compliance work detection is required, that is, an area to determine whether the target object's work meets the preset regulations. For example, when a worker enters the working area, it is necessary to detect whether the worker wears a safety helmet and a safety vest. The possible working range 102 also includes a preparation area 104, which is an area for entering the working area from outside the working area. Usually, the target object needs to perform relevant preparations for the work in the preparation area before performing the work, such as wearing a safety helmet, wearing a safety vest, activating certain safety protection devices (such as turning on the lighting on the safety helmet, hanging the safety hook in a fixed place when climbing), etc. Four cameras 105 are set in this scene, and images in the possible working range 102 are captured from different shooting angles.

[0083] First, a video analysis method of the present application embodiment is described by taking a camera as an example. Figure 2 , a video analysis method provided by an embodiment of the present application includes:

[0084] Step 201, obtaining an image captured by a camera.

[0085] Here, video data of the target object within a possible working range is collected by a camera, thereby obtaining continuous multiple-frame images collected by the camera.

[0086] Step 202, perform element detection on the image to obtain the elements present in each frame of the image, wherein the elements include area elements, main elements and determination elements; the area elements include at least the working area; the main elements are the target objects; the determination elements are elements for determining whether the working is compliant, and the determination elements include a compliance status and a violation status.

[0087] Here, the embodiment of the present application performs element detection on the image captured by the camera, and the elements specifically include area elements, subject elements and determination elements. The detection of various elements is described below.

[0088] The regional elements usually include working areas and may also include preparation areas, etc. In a specific scenario, the working area and preparation area are usually fixed areas with unchanged positions set in advance. In this case, the working area and preparation area in the image can be determined according to the preset fixed areas. Figure 3-4An example of a fixed work area and preparation area is shown, and the example includes two parking spaces, namely parking space 21 and parking space 22. The vehicle to be worked on is always parked in a fixed posture on parking space 22, so the roof area as the work area and the platform of the climbing ladder adjacent to the work area are both fixed areas, and the shooting area of ​​each camera is also a fixed area. Therefore, the work area and preparation area in the image can be determined by the correspondence between the predetermined work area / preparation area and the shooting area of ​​the camera.

[0089] In other scenarios, the working area and the preparation area may both be non-fixed areas. In this case, the image semantic segmentation technology is used to perform image segmentation on the regional elements to determine the working area and the preparation area in the image. For example, image segmentation is performed based on the image features in each regional element to determine the working area and the preparation area. Figure 5-6 An example of a non-fixed work area and preparation area is shown, which includes two parking spaces, namely parking space 31 and parking space 32. The vehicle to be worked on may be parked in parking space 31 or parking space 32. Therefore, the roof area of ​​the work area and the platform of the climbing ladder adjacent to the work area are both non-fixed areas. For example, Figure 6 In some examples, the vehicle is parked in parking space 32, while in other examples, the vehicle may be parked in parking space 31. At this time, the image can be segmented according to the image features of the working area (such as the image features of the vehicle), and the vehicle and the roof part can be segmented from the image, and then the roof part can be determined as the working area; the image can be segmented according to the image features of the preparation area (such as the image features of the climbing ladder), and the climbing ladder and the climbing ladder surface can be segmented from the image, and then the climbing ladder surface can be determined as the preparation area, and so on.

[0090] In the embodiment of the present application, the subject element is the target object for compliance operation detection, such as workers, etc. There can be one or more judgment elements, and the judgment element is an element used to determine whether the operation is compliant, and the judgment element includes a compliance state and a violation state. The type of the judgment element can be selected according to the specific operation scene and operation specifications, including but not limited to one or more of the following: the wearing state of the helmet; the wearing state of the safety vest; the on and off state of the lighting on the helmet; the hanging state of the safety hook, etc.

[0091] Taking wearing a safety helmet and wearing a safety vest as an example, the wearing state of the safety helmet corresponds to a determination factor, which includes a compliance state (the worn state of the safety helmet being worn) and a violation state (the not worn state of the safety helmet not being worn). Similarly, the wearing state of the safety vest corresponds to another determination factor, which includes a compliance state (the worn state of the safety vest being worn) and a violation state (the not worn state of the safety vest not being worn). That is to say, at this time, the determination factor is used to represent the wearing state of the target object of the target object in the image, for example, the worn state or the not worn state. The target object is an item that needs to be worn during compliant operations.

[0092] For another example, for the on / off state of the headlight on a helmet, the compliance state of the determination factor is the on state, and the violation state is the off state. For the hanging state of a safety hook, the compliance state of the determination factor is hanging in a fixed place, and the violation state is not hanging in a fixed place, and so on.

[0093] Specifically, the main elements of the image can be detected by target object recognition technology to determine whether the image has main elements, and when the main elements exist in the image, the target items corresponding to the determination elements (such as helmets, safety vests, safety hooks and fixings, etc.) can be further detected by the object recognition technology to obtain the determination elements corresponding to the image. Taking wearing a helmet as an example, when it is identified that the main element in the image is wearing a helmet, a determination element corresponding to the wearing state of the helmet is generated, and the determination element is in a compliant state (worn state of wearing a helmet); conversely, when it is identified that the main element in the image is not wearing a helmet, a determination element corresponding to the wearing state of the helmet is generated, and the determination element is in a non-compliant state (not worn state of not wearing a helmet).

[0094] Step 203, when the operation area and the main element are detected, the main element is tracked, and after the main element enters the operation area, whether the operation of the main element is compliant is determined based on the judgment elements in the multiple frames of images.

[0095] Here, if the working area and the main elements are detected from the image, the embodiment of the present application can track the main elements. In this way, after the main elements enter the working area, the judgment elements in multiple frames of images can be obtained based on the element detection results of the image, and then it can be determined whether the operation of the main elements is compliant.

[0096] Specifically, for each judgment factor, a queue consisting of judgment factors corresponding to each frame of the multiple frames of images is obtained; whether the judgment factor meets the job requirements is determined based on whether the proportion of compliant status judgment factors in the queue is greater than a preset first threshold; when any judgment factor does not meet the job requirements, it is determined that the main factor operation is non-compliant; when all judgment factors meet the job requirements, it is determined that the main factor operation is compliant.

[0097] Taking the wearing of safety helmets and safety vests as an example, for each target item (safety helmet, safety vest), a queue consisting of the determination elements corresponding to the target item in the multiple frames of images is obtained; and whether the subject element wears the target item is determined according to whether the proportion of the worn state in the queue is greater than the preset first threshold. For example, when the proportion of the worn state in the queue is greater than the first threshold, it is determined that the subject element has worn the target item, that is, the determination element corresponding to the target item meets the operation requirements; when the proportion of the worn state in the queue is less than or equal to the first threshold, it is determined that the subject element does not wear the target item, that is, the determination element corresponding to the target item does not meet the operation requirements. Alternatively, when the proportion of the unworn state in the queue is greater than or equal to the second threshold, it is determined that the subject element does not wear the target item, that is, the determination element corresponding to the target item does not meet the operation requirements; when the proportion of the unworn state in the queue is less than the second threshold, it is determined that the subject element has worn the target item, that is, the determination element corresponding to the target item meets the operation requirements. Here, the first threshold and the second threshold can be set according to the specific environment. For example, the first threshold can be a value of about 0.9, and the second threshold can be a value of about 0.1.

[0098] Then, if it is determined based on the multiple frames of images captured by the camera that the subject element is not wearing any target item, then it is determined that the subject element operation is not compliant; if it is determined based on the multiple frames of images captured by the camera that the subject element has worn all target items, then it is determined that the subject element operation is compliant. All target items are determined in advance according to the requirements of compliant operations.

[0099] Through the above steps, the embodiment of the present application performs element detection based on multiple frames of images captured by the camera, realizes compliance operation detection in complex dynamic scenes, and can improve the safety of production operations. The above scheme of the embodiment of the present application is a general visual violation detection scheme that can be applied to a variety of operation scenarios and has good adaptability and reliability.

[0100] Optionally, in the above step 203, the embodiment of the present application tracks the subject element to obtain a tracking trajectory. According to the tracking trajectory, after determining that the subject element enters the operation area through the preparation area, the step of determining whether the operation of the subject element is compliant based on the judgment elements in the multiple frames of images is performed, so as to avoid the misjudgment that the subject element appears in the operation area without passing through the preparation area.

[0101] Furthermore, the embodiment of the present application can also execute the step of determining whether the operation of the subject element is compliant based on the judgment elements in the multiple frames of images only when the time length of the subject element entering the work area is greater than the preset time length threshold. This can improve the accuracy of the judgment result. Therefore, the subject element usually moves faster in the initial stage of entering the work area, and the accuracy of the image detection result at this time is lower.

[0102] The above embodiment takes one camera as an example to introduce the video analysis method of the embodiment of the present application. The following further describes the processing method when multiple cameras (greater than or equal to 2 cameras) are used.

[0103] When multiple cameras are used, the same processing is performed for each camera according to the above steps 201 to 202, that is, the image captured by the camera is obtained, and the element detection is performed on the image captured by the camera to obtain the elements existing in each frame of the image. Then, in step 203, when the operation area and the main element are detected in the image captured by the camera, the main element is tracked, and after the main element enters the operation area, it is determined whether the operation of the main element is compliant according to the determination elements in the multiple frames of images, which specifically includes:

[0104] (1) For each judgment element, determine whether the judgment element meets the operation requirements according to the following steps: for each camera, obtain a queue consisting of judgment elements corresponding to each frame of multiple frames of images captured by the camera, where the multiple frames of images captured by different cameras are collected synchronously; determine a judgment result corresponding to the camera according to whether the proportion of compliance status judgment elements in the queue is greater than a preset first threshold, and the judgment result is whether the operation requirements are met or not met; determine whether the judgment element meets the operation requirements according to a judgment result that accounts for a larger proportion of the judgment results corresponding to the multiple cameras;

[0105] (2) when any of the determination elements does not meet the operation requirements, the operation of the subject element is determined to be non-compliant;

[0106] (3) When all the determination factors meet the operation requirements, the operation of the subject factor is determined to be compliant.

[0107] The following will continue to explain in detail using the examples of wearing a safety helmet and wearing a safety vest.

[0108] For each target item (such as a helmet and a safety vest), determine whether the subject element is wearing the target item according to the following steps: for each camera, obtain a queue consisting of the judgment elements corresponding to the target item in the multiple frames of images collected by the camera, where the multiple frames of images collected by different cameras are collected synchronously. Then, according to whether the proportion of the wearing state in the queue is greater than a preset first threshold, determine the judgment result corresponding to the camera, wherein the judgment result is whether the target item is worn or not worn; according to the judgment result with a larger proportion in the judgment results corresponding to the multiple cameras, determine whether the subject element is wearing the target item. For example, if there are a total of 4 cameras, the judgment results corresponding to 3 cameras are that the target item 1 is worn, and the judgment result corresponding to the remaining 1 camera is that the target item 1 is not worn, then the judgment result with a larger proportion is that the target item 1 is worn, and at this time it is determined that the subject element is wearing the target item 1; for another example, if the judgment results corresponding to 3 cameras are that the target item 2 is not worn, and the judgment result corresponding to the remaining 1 camera is that the target item 2 is worn, then the judgment result with a larger proportion is that the target item 2 is not worn, and at this time it is determined that the subject element is not wearing the target item 2.

[0109] Then, when the subject element does not wear any target item, it is determined that the subject element operation is not compliant; when the subject element has worn all target items, it is determined that the subject element operation is compliant.

[0110] When the number of the multiple cameras is an even number, it may happen that among the judgment results corresponding to the multiple cameras, the proportion of the judgment results that meet the operation requirements and the judgment results that do not meet the operation requirements is the same. At this time, the embodiment of the present application can further determine whether the subject element is wearing the target item in the following manner.

[0111] Specifically, in step 202 of the present application embodiment, when detecting the target object in the image through the target object recognition technology, the confidence of the judgment factor corresponding to the image is further obtained, that is, the confidence of the violation state and the confidence of the compliance state of the judgment factor. In this way, in the judgment results corresponding to the multiple cameras, when the proportion of the judgment results that meet the operation requirements and the judgment results that do not meet the operation requirements is the same:

[0112] (1) For a camera whose judgment result is that it meets the job requirements, calculate the average confidence of the compliance status in the judgment factor corresponding to the multiple frames of images captured by the camera to obtain a first average confidence corresponding to the camera; sum the first average confidences corresponding to all cameras whose judgment results are that they meet the job requirements to obtain a first score.

[0113] For example, in the example of continuing to wear a safety helmet or a safety vest, for the camera whose judgment result is that the target item (such as a safety helmet or a safety vest) has been worn, the average confidence value of the worn state in the multiple frames of images captured by the camera can be calculated to obtain a first average confidence value corresponding to the camera; then, the first average confidence values ​​corresponding to all cameras whose judgment result is that the target item has been worn can be summed to obtain a first score.

[0114] (2) For cameras whose judgment results are that they do not meet the job requirements, calculate the average confidence of the violation status in the judgment factors corresponding to the multiple frames of images captured by the camera to obtain a second average confidence corresponding to the camera; sum the second average confidences corresponding to all cameras whose judgment results are that they do not meet the job requirements to obtain a second score.

[0115] For example, in the example of continuing to wear a safety helmet or a safety vest, for a camera whose judgment result is that the target item (such as a safety helmet or a safety vest) is not being worn, the average confidence value of the not-wearing state in the multiple frames of images captured by the camera can be calculated to obtain a second average confidence value corresponding to the camera; the second average confidence values ​​corresponding to all cameras whose judgment result is that the target item is not being worn can be summed to obtain a second score.

[0116] (3) Then, based on the size relationship between the first score and the second score, determine whether the determination factor meets the job requirements, wherein when the first score is greater than the second score, determine that the determination factor meets the job requirements; when the first score is not greater than the second score, determine that the determination factor does not meet the job requirements.

[0117] For example, in the example of continuing to wear a helmet or a safety vest, whether the subject element is wearing the target item (such as a helmet or a safety vest) can be determined based on the magnitude relationship between the first score and the second score. When the first score is greater than the second score, it is determined that the subject element has worn the target item; when the first score is not greater than the second score, it is determined that the subject element has not worn the target item.

[0118] Through the above steps, the embodiment of the present application can detect compliance operations of the target object based on images captured by multiple cameras. Since multiple cameras can provide different shooting angles, more comprehensive image information can be obtained, thereby improving the accuracy of compliance operation detection results and better ensuring production safety.

[0119] The above method of the embodiment of the present application can generate an alarm that any of the elements do not meet the operation requirements and / or the operation of the main element is not compliant when it is determined that any of the determination elements do not meet the operation requirements and / or the operation of the main element is not compliant. The alarm can be one or more forms such as sound, text, and light. Through the alarm, the target object and / or the management personnel are notified of the operation violation of the main element, so as to facilitate timely operation control and improve production safety.

[0120] To help better understand the above embodiments, the video analysis method of the embodiment of the present application is further described below through more specific examples. The following examples are mainly described by wearing a safety helmet and a safety vest as an example.

[0121] like Figure 7 As shown, the process of this example mainly includes: collecting on-site data at a certain frequency through the camera, and the collected video stream is disassembled into continuous images through decoding. The continuous images are analyzed using object detection algorithms, object tracking algorithms and image semantic segmentation algorithms, and the results of the analysis are input into the spatiotemporal logic judgment method to determine whether there is a violation. The process of this example can be applied to cameras set at different positions and shooting angles in the same scene, and multi-camera weighted voting joint judgment is performed. The following text will take an aerial operation as an example to illustrate.

[0122] According to the business scenario and business process, it is necessary to first identify the key areas in the business: possible work scope, operation area, and preparation area, as mentioned above. Figure 1 As shown. The operation area refers to the area where illegal operations can only occur, or only illegal operations occurring in this area are considered business illegal operations. The possible working range refers to the range where the operation area may exist, and its range may be consistent with the operation range. The preparation area refers to the area from outside the operation area to the operation area. Taking the climbing operation on the vehicle body as an example, the operation area refers to the vehicle body (roof), and safety helmets and safety vests must be worn in this area, otherwise it is a violation. The possible working range refers to the range where the vehicle body may be parked. The preparation area is the location of the climbing ladder, and workers can only climb through this area.

[0123] For cameras that capture business scenarios, set them up based on the principle of fully covering the possible working range. For reference, Figure 3-4 shown.

[0124] For some business scenarios, the operation area and preparation area are not fixed. Figure 5 In this case, image semantic segmentation technology can be used to segment the regional elements. Dynamically set the working area and preparation area, such as Figure 6 shown.

[0125] When performing feature detection, common image object recognition technology can be used to detect features in the image. Features can be divided into regional features, subject features, and judgment features. Regional features refer to features related to the work area, such as the work vehicle body. Subject features refer to the subject object for violation determination, such as workers. Judgment features refer to features related to violation determination, such as the wearing status of safety helmets and safety vests.

[0126] This example uses a common image object tracking algorithm to track the main elements in the image, thereby obtaining their moving lines (or trajectories), such as Figure 8 As shown. For example, the movement of a worker is tracked. For the subject being tracked, a decision factor queue is established for the state of the decision factors attached to it. For example, a queue of 100 consecutive frames stores the state of whether a worker subject is wearing a safety helmet when moving in these 100 frames. A representation of the decision factor queue J is as follows:

[0127] J={(j1,c1),(j2,c2),…,(j i ,c i ),…,(j N ,c N )}

[0128] Among them, j i = 0 or 1, indicating whether it is in violation, for example, 0 indicates compliance and 1 indicates violation; c i Indicates the confidence level of the violation, 1-c i It represents the confidence level of compliance, and N is the queue length of the decision factor queue.

[0129] Assume that a decision factor queue of length 100 is: {(1,0.8),(1,0.9),(0,0.9),…,(1,0.9)}, where j i =1 is 18, assuming that the threshold value of the violation ratio is 0.1, since 18 / 100=0.18, and 0.18 is greater than 0.1, the subject is judged to have violated the rules based on the 100 frames of images.

[0130] In order to reduce misjudgment and improve the robustness of the method, this example uses the aforementioned feature detection results, region segmentation results, and subject tracking results as inputs to design a violation judgment method based on spatiotemporal logic. Fig. 9 As shown, it mainly includes the following steps:

[0131] (1) Determine whether the regional element exists: If it does, continue with the following judgment; otherwise, continue with element detection.

[0132] (2) Determine whether the main element has entered the preparation area: If so, continue with the following judgment; otherwise, continue with element detection.

[0133] (3) Determine whether the subject element has entered the working area: If so, collect each determination element in sequence, obtain the determination element queue of the current subject, and continue the following determination; otherwise, continue the element detection.

[0134] (4) Determine whether the time (number of frames) that the main element enters the working area is greater than the set threshold (e.g., 200 frames): If yes, continue with the following judgment; otherwise, continue with element detection

[0135] (5) Determine whether the proportion of illegal elements in the judgment element 1 queue exceeds the set threshold (e.g., 90%): If yes, an alarm is issued regarding element 1; otherwise, proceed to the judgment of the next judgment element queue

[0136] (6) The same judgment is performed for elements 2 to n (n is the total number of judgment elements).

[0137] (7) If no alarm is generated in any of the decision factor queues, the main operation is judged to be compliant.

[0138] Through the above steps, this example continuously detects whether the main operation violates the rules and generates corresponding alarms when violations are detected to improve the safety of the operation.

[0139] In order to ensure the robustness of the above-mentioned video analysis method, another example uses cameras with different shooting angles to monitor the same scene, and performs weighted voting on the analysis results of different cameras. The following takes 4 cameras as an example for explanation, in which each judgment factor is judged whether it is in violation based on the image collected by each camera, and the specific judgment method is the same as above. After obtaining the judgment results of each camera for the same judgment factor, if the number of cameras with violation judgment results and compliance judgment results is not equal, the judgment results of the majority of cameras are used as the final judgment results of the judgment factor. For example, in the judgment result of a certain judgment factor, assuming that 3 of the 4 cameras are judged to be compliant and 1 camera is judged to be illegal, the final result of the judgment factor is compliant. When the number of cameras with violation judgment results and compliance judgment results is equal, weighted voting can be performed to determine the final judgment result based on the voting results. For example, assuming that the judgment factor queue corresponding to a judgment factor of a certain camera is:

[0140] J={(j1,c1),(j2,c2),…,(j i ,c i ),…,(j N ,c N )}

[0141] Among them, j i = 0 or 1, indicating whether it is in violation, for example, 0 indicates compliance and 1 indicates violation; c i Indicates the confidence level of the violation, 1-c i It represents the confidence level of compliance, and N is the queue length of the decision factor queue.

[0142] If the camera's judgment result for the judgment factor is a violation, the average confidence level of the violation judgment result is calculated as follows:

[0143] If the camera's judgment result for the judgment factor is compliant, the average confidence level of the compliant judgment result is calculated as follows:

[0144] Through the above method, the average confidence of the determination results of each camera can be obtained.

[0145] Then, based on the average confidence of the judgment results of each camera, the weighted voting is defined as follows:

[0146]

[0147] Among them, T - is the overall score of the violation, T + The overall score for compliance is NC - The number of cameras that violated the regulations, NC + is the number of cameras that comply with the regulations. That is, the average confidence of all cameras that are judged as non-compliant is accumulated to obtain T - ; Accumulate the average confidence of all cameras that are judged as compliant to obtain T + .

[0148] Then, by comparing T - With T + size, and obtain the final judgment result. - Greater than T + For example, if 2 of the 4 cameras are judged as compliant and 2 are judged as illegal, multi-camera weighted voting is performed, T - =1.90 and T + =1.41 is considered a violation.

[0149] Combine the following Figure 10 to Figure 13 , to illustrate this, take a factory as an example in which it determines whether the work done by workers on a car body is in violation of regulations.

[0150] Assume that the factory stipulates that workers (main element) must wear safety helmets (corresponding to judgment element 1) and safety vests (corresponding to judgment element 2) all the time when working on the car body (operation area). In this case, they need to use a ladder (preparation area) to enter the operation area.

[0151] Assume that the camera used for acquisition is 25 frames / second, and set the judgment element queue length to 100 (4s). First, use element detection, main element tracking and region segmentation technologies to process the decoded continuous image sequence. When the tracking trajectory of the main element enters the working area through the preparation area, the judgment process begins. This can effectively eliminate the misjudgment of appearing in the working area without passing through the preparation area, such as Fig.10 shown.

[0152] When the main element enters the work area and the time spent in the work area meets the threshold (set to 4s), it starts to judge whether there is a violation. The 4s threshold is set to reduce the possibility of false detection, because in certain cases, there may be a stable period of state when entering the work area from the preparation area. For example, when crossing from the ladder to the roof, the main element moves faster, and the probability of misjudgment of image detection during this period increases. The violation ratio is calculated for the queues of different judgment elements, and an alarm is issued when the ratio is greater than the threshold (set to 10%). When all the judgment element queues have no violations, the main element operation is judged to be compliant. The 10% threshold is set to eliminate misjudgments. When there are 100 judgment element states in the queue, if one of the judgment elements is mistakenly judged by the algorithm from compliance to violation, then the violation ratio of the entire queue is 1%, which is lower than the set threshold of 10%, thereby effectively eliminating misjudgments. Fig.11 This is a case of compliance judgment, where the subject element (worker) wears a safety helmet and a safety vest. Fig.12 This is a case where determination factor 1 (corresponding to the helmet) is determined to be a violation. Fig.13 This is a case where both determination factor 1 (corresponding to the safety helmet) and determination factor 2 (corresponding to the safety vest) are determined to be violations.

[0153] Please refer to Fig.14 , a structure of a video analysis system provided in an embodiment of the present application includes:

[0154] An image acquisition module 141 is used to acquire images captured by a camera;

[0155] The element detection module 142 is used to perform element detection on the image to obtain elements existing in each frame of the image, wherein the elements include regional elements, main elements and determination elements; the regional elements at least include the operation area, and the operation area is the area where compliance operation detection is required; the main elements are target objects; the determination elements are elements for determining whether the operation is compliant, and the determination elements include compliance status and violation status;

[0156] The compliance judgment module 143 is used to track the main element when the operation area and the main element are detected, and after the main element enters the operation area, determine whether the operation of the main element is compliant based on the judgment factors in multiple frames of images.

[0157] Through the above modules, the embodiments of the present application can improve the safety of production operations.

[0158] Optionally, the area element further includes a preparation area, which is an area from outside the working area to the working area; the element detection module is further used to:

[0159] When both the working area and the preparation area are pre-set fixed areas, determining the working area and the preparation area in the image according to the pre-set fixed areas;

[0160] When both the working area and the preparation area are non-fixed areas, image segmentation is performed on regional elements using image semantic segmentation technology to determine the working area and the preparation area in the image.

[0161] Optionally, in the above system, the compliance judgment module is also used to execute the step of determining whether the operation of the main element is compliant based on the judgment elements in the multiple frames of images when the main element enters the operating area through the preparation area and the time length in the operating area is greater than a preset time length threshold based on the tracking trajectory obtained by tracking the main element.

[0162] Optionally, the element detection module is also used to detect the main elements of the image through target object recognition technology to determine whether the main elements exist in the image, and when the main elements exist in the image, further detect the target objects in the image through object recognition technology based on the target objects corresponding to the determination elements to obtain the determination elements corresponding to the image.

[0163] Optionally, the compliance judgment module is further used to:

[0164] For each determination factor, a queue consisting of determination factors corresponding to each frame of the multiple images is obtained; and whether the determination factor meets the operation requirements is determined according to whether the proportion of compliance status determination factors in the queue is greater than a preset first threshold value;

[0165] When any of the determination elements does not meet the operation requirements, it is determined that the operation of the main element is non-compliant;

[0166] When all the determination factors meet the operation requirements, it is determined that the operation of the main elements is compliant.

[0167] Optionally, there are multiple cameras;

[0168] The compliance judgment module is further used to:

[0169] For each determination factor, determine whether the determination factor meets the job requirements according to the following steps: for each camera, obtain a queue consisting of determination factors corresponding to each frame of multiple frames of images captured by the camera; determine a determination result corresponding to the camera according to whether the proportion of compliance status determination factors in the queue is greater than a preset first threshold, and the determination result is whether the job requirements are met or not met; determine whether the determination factor meets the job requirements according to a determination result that accounts for a larger proportion of the determination results corresponding to the multiple cameras;

[0170] When any of the determination elements does not meet the operation requirements, it is determined that the operation of the main element is non-compliant;

[0171] When all the determination factors meet the operation requirements, it is determined that the operation of the main elements is compliant.

[0172] Optionally, the element detection module is further used to further obtain the confidence of the determination element corresponding to the image when detecting the target object in the image through the object recognition technology;

[0173] The compliance judgment module is further configured to: when, among the judgment results corresponding to the multiple cameras, the proportion of the judgment results that meet the operation requirements is the same as the proportion of the judgment results that do not meet the operation requirements:

[0174] For a camera whose judgment result is that the operation requirements are met, the average confidence of the compliance status in the judgment factor corresponding to the multiple frames of images collected by the camera is calculated to obtain a first average confidence corresponding to the camera; the first average confidence corresponding to all cameras whose judgment results are that the operation requirements are met is summed to obtain a first score;

[0175] For a camera whose judgment result is that the operation requirements are not met, the average confidence of the violation status in the judgment factor corresponding to the multiple frames of images collected by the camera is calculated to obtain a second average confidence corresponding to the camera; the second average confidence corresponding to all cameras whose judgment results are that the operation requirements are not met is summed to obtain a second score;

[0176] When the first score is greater than the second score, determining that the determination factor meets the operation requirement;

[0177] When the first score is not greater than the second score, it is determined that the determination factor does not meet the operation requirement.

[0178] Optionally, the above system further includes:

[0179] The alarm module is used to generate an alarm that the subject element is not wearing any target item and / or the subject element is not operating in compliance with regulations when the subject element is not wearing any target item.

[0180] It should be noted that the various systems provided in the above embodiments are devices corresponding to the above video analysis methods, and the implementation methods in the above embodiments are applicable to the embodiments of the device, and can also achieve the same technical effects. The above device provided in the embodiment of the present application can implement all the method steps implemented in the above method embodiment, and can achieve the same technical effects, and the parts and beneficial effects that are the same as those in the method embodiment will not be specifically described here.

[0181] Please refer to Fig.15 , a schematic diagram of the structure of another video analysis system provided in an embodiment of the present application, the device includes: a processor 1501, a transceiver 1502, a memory 1503, a user interface 1504 and a bus interface.

[0182] In the embodiment of the present application, the device also includes: a program stored in the memory 1503 and executable on the processor 1501 .

[0183] The transceiver 1502 is used to send and receive data under the control of the processor;

[0184] The processor 1501 is configured to read the computer program in the memory and perform the following operations:

[0185] Get the image captured by the camera;

[0186] Performing element detection on the image to obtain elements existing in each frame of the image, wherein the elements include regional elements, main elements and determination elements; the regional elements at least include an operation area, and the operation area is an area that needs to be detected for compliance operations; the main elements are target objects; the determination elements are elements for determining whether the operation is compliant, and the determination elements include compliance status and violation status;

[0187] When the operation area and the main element are detected, the main element is tracked, and after the main element enters the operation area, it is determined whether the operation of the main element is compliant based on the determination elements in the multiple frames of images.

[0188] It can be understood that in the embodiment of the present application, when the computer program is executed by the processor 1501, the various processes of the above-mentioned video analysis method embodiment can be implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0189] exist Fig.15 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 1501 and various circuits of memory represented by memory 1503 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 1502 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 1504 may also be an interface capable of externally and internally connecting required devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.

[0190] The processor 1501 is responsible for managing the bus architecture and general processing, and the memory 1503 can store data used by the processor 1501 when performing operations.

[0191] It should be noted that the device in this embodiment is a device corresponding to the above-mentioned video analysis method, and the implementation methods in the above-mentioned embodiments are all applicable to the embodiments of the device, and can also achieve the same technical effects. In the device, the transceiver 1502 and the memory 1503, as well as the transceiver 1502 and the processor 1501 can be connected through the bus interface communication, the function of the processor 1501 can also be implemented by the transceiver 1502, and the function of the transceiver 1502 can also be implemented by the processor 1501. It should be noted that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effects, and the parts and beneficial effects that are the same as the method embodiment in this embodiment will not be specifically repeated here.

[0192] In some embodiments of the present application, a computer-readable storage medium is further provided, on which a program is stored, and when the program is executed by a processor, the following steps are implemented:

[0193] Get the image captured by the camera;

[0194] Performing element detection on the image to obtain elements existing in each frame of the image, wherein the elements include regional elements, main elements and determination elements; the regional elements at least include an operation area, and the operation area is an area that needs to be detected for compliance operations; the main elements are target objects; the determination elements are elements for determining whether the operation is compliant, and the determination elements include compliance status and violation status;

[0195] When the operation area and the main element are detected, the main element is tracked, and after the main element enters the operation area, it is determined whether the operation of the main element is compliant based on the determination elements in the multiple frames of images.

[0196] When the program is executed by the processor, it can implement all the implementation methods in the above-mentioned video analysis method and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0197] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0198] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0199] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0200] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0201] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0202] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0203] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A video analysis method, It is characterized in that include: Get the image captured by the camera; Performing element detection on the image to obtain elements existing in each frame of the image, wherein the elements include regional elements, main elements and determination elements; the regional elements at least include an operation area, and the operation area is an area that needs to be detected for compliance operations; the main elements are target objects; the determination elements are elements for determining whether the operation is compliant, and the determination elements include compliance status and violation status; When the operation area and the main element are detected, the main element is tracked, and after the main element enters the operation area, it is determined whether the operation of the main element is compliant based on the determination elements in the multiple frames of images.

2. The method according to claim 1, It is characterized in that The area elements also include a preparation area, which is an area for entering the working area from outside the working area; Performing element detection on the image includes: When both the working area and the preparation area are pre-set fixed areas, determining the working area and the preparation area in the image according to the pre-set fixed areas; When both the working area and the preparation area are non-fixed areas, image segmentation is performed on regional elements using image semantic segmentation technology to determine the working area and the preparation area in the image.

3. The method according to claim 2, It is characterized in that Also includes: According to the tracking trajectory obtained by tracking the main element, when the main element enters the operation area through the preparation area and the time length in the operation area is greater than the preset time length threshold, the step of determining whether the operation of the main element is compliant based on the judgment elements in the multiple frames of images is executed.

4. The method according to claim 2 or 3, It is characterized in that Performing element detection on the image further includes: The target object recognition technology is used to detect the main elements of the image to determine whether the main elements exist in the image. When the main elements exist in the image, the target objects corresponding to the determination elements are further detected in the image through the object recognition technology to obtain the determination elements corresponding to the image.

5. The method according to claim 4, It is characterized in that Determining whether the operation of the subject element is compliant according to the determination elements in the multiple frames of images includes: For each determination factor, a queue consisting of determination factors corresponding to each frame of the multiple images is obtained; and whether the determination factor meets the operation requirements is determined according to whether the proportion of compliance status determination factors in the queue is greater than a preset first threshold value; When any of the determination elements does not meet the operation requirements, it is determined that the operation of the main element is non-compliant; When all the determination factors meet the operation requirements, it is determined that the operation of the main elements is compliant.

6. The method according to claim 4, wherein It is characterized in that There are multiple cameras; Determining whether the operation of the subject element is compliant according to the determination elements in the multiple frames of images, including: for each determination element, determining whether the determination element meets the operation requirements according to the following steps: for each camera, obtaining a queue consisting of determination elements corresponding to each frame of the multiple frames of images captured by the camera; determining a determination result corresponding to the camera according to whether the proportion of compliant state determination elements in the queue is greater than a preset first threshold, the determination result being that the operation requirements are met or not met; determining whether the determination element meets the operation requirements according to a determination result that accounts for a larger proportion of the determination results corresponding to the multiple cameras; When any of the determination elements does not meet the operation requirements, it is determined that the operation of the main element is non-compliant; When all the determination factors meet the operation requirements, it is determined that the operation of the main elements is compliant.

7. The method according to claim 6, characterized in that: When detecting the target object in the image through the object recognition technology, further obtaining the confidence of the determination factor corresponding to the image; If, among the judgment results corresponding to the multiple cameras, the proportion of the judgment results that meet the operation requirements is the same as the proportion of the judgment results that do not meet the operation requirements, the method further includes: For a camera whose judgment result is that the operation requirements are met, the average confidence of the compliance status in the judgment factor corresponding to the multiple frames of images collected by the camera is calculated to obtain a first average confidence corresponding to the camera; the first average confidence corresponding to all cameras whose judgment results are that the operation requirements are met is summed to obtain a first score; For a camera whose judgment result is that the operation requirements are not met, the average confidence of the violation status in the judgment factor corresponding to the multiple frames of images collected by the camera is calculated to obtain a second average confidence corresponding to the camera; the second average confidence corresponding to all cameras whose judgment results are that the operation requirements are not met is summed to obtain a second score; When the first score is greater than the second score, determining that the determination factor meets the operation requirement; When the first score is not greater than the second score, it is determined that the determination factor does not meet the operation requirement.

8. The method according to claim 7, It is characterized in that Also includes: When any of the determination elements does not meet the operation requirements, an alarm is generated that any of the elements does not meet the operation requirements and / or the operation of the main element is not compliant.

9. A video analysis system, It is characterized in that include: An image acquisition module is used to acquire images captured by a camera; An element detection module is used to perform element detection on the image to obtain elements existing in each frame of the image, wherein the elements include regional elements, main elements and determination elements; the regional elements at least include an operation area, and the operation area is an area that needs to be detected for compliance operations; the main elements are target objects; the determination elements are elements for determining whether the operation is compliant, and the determination elements include compliance status and violation status; The compliance judgment module is used to track the main element when the operation area and the main element are detected, and after the main element enters the operation area, determine whether the operation of the main element is compliant based on the judgment elements in multiple frames of images.

10. The system according to claim 9, It is characterized in that The regional elements also include a preparation area, which is an area from outside the working area to the working area; the element detection module is also used to: When both the working area and the preparation area are pre-set fixed areas, determining the working area and the preparation area in the image according to the pre-set fixed areas; When both the working area and the preparation area are non-fixed areas, image segmentation is performed on regional elements using image semantic segmentation technology to determine the working area and the preparation area in the image.

11. The system according to claim 10, It is characterized in that The element detection module is also used to detect the main elements of the image through target object recognition technology to determine whether the image has the main elements, and when the main elements exist in the image, further detect the target objects corresponding to the determination elements in the image through object recognition technology to obtain the determination elements corresponding to the image.

12. The system according to claim 11, It is characterized in that The compliance judgment module is further used to: For each determination factor, a queue consisting of determination factors corresponding to each frame of the multiple images is obtained; and whether the determination factor meets the operation requirements is determined according to whether the proportion of compliance status determination factors in the queue is greater than a preset first threshold value; When any of the determination elements does not meet the operation requirements, it is determined that the operation of the main element is non-compliant; When all the determination factors meet the operation requirements, it is determined that the operation of the main elements is compliant.

13. The system according to claim 11, characterized in that , There are multiple cameras; The compliance judgment module is further used to: For each determination factor, determine whether the determination factor meets the job requirements according to the following steps: for each camera, obtain a queue consisting of determination factors corresponding to each frame of multiple frames of images captured by the camera; determine a determination result corresponding to the camera according to whether the proportion of compliance status determination factors in the queue is greater than a preset first threshold, and the determination result is whether the job requirements are met or not met; Determining whether the determination factor meets the operation requirement according to a determination result that accounts for a larger proportion among the determination results corresponding to the multiple cameras; When any of the determination elements does not meet the operation requirements, it is determined that the operation of the main element is non-compliant; When all the determination factors meet the operation requirements, it is determined that the operation of the main elements is compliant.

14. The system according to claim 13, It is characterized in that The element detection module is further used to further obtain the confidence of the determination element corresponding to the image when detecting the target object in the image through the object recognition technology; The compliance judgment module is further configured to: when, among the judgment results corresponding to the multiple cameras, the proportion of the judgment results that meet the operation requirements is the same as the proportion of the judgment results that do not meet the operation requirements: For a camera whose judgment result is that it meets the operation requirements, calculate the average confidence of the compliance status in the judgment factor corresponding to the multiple frames of images collected by the camera to obtain a first average confidence corresponding to the camera; Sum the first average confidences corresponding to all cameras whose judgment results are that they meet the task requirements to obtain a first score; For a camera whose judgment result is that it does not meet the operation requirements, calculate the average confidence of the violation status in the judgment factor corresponding to the multiple frames of images collected by the camera to obtain a second average confidence corresponding to the camera; Sum the second average confidences corresponding to all cameras whose judgment results are that they do not meet the operation requirements to obtain a second score; When the first score is greater than the second score, determining that the determination factor meets the operation requirement; When the first score is not greater than the second score, it is determined that the determination factor does not meet the operation requirement.

15. The system according to claim 14, It is characterized in that Also includes: The alarm module is used to generate an alarm that the subject element is not wearing any target item and / or the subject element is not operating in compliance with regulations when the subject element is not wearing any target item.