Security detection method and device, medium, electronic equipment and program product
By introducing dual processing of posture detection and machine learning models in action detection, the problem of insufficient robustness for unseen objects and environment detection in the prior art is solved, and the effectiveness of action detection and reliability of safety detection are improved.
Patent Information
- Application Number
- CN202411836497.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-12
- Publication Date
- 2025-03-04
AI Technical Summary
The existing action detection model lacks robustness when detecting unseen objects and environments, resulting in a decrease in the effectiveness and confidence of action detection, thereby reducing the reliability and efficiency of safety detection.
By performing pose detection on each frame in the video to be detected, frames that may contain target actions of the target object are filtered as keyframes, and these keyframes are processed based on the machine learning model to determine whether the target actions of the target object are indeed included.
It improves the generalization ability of unseen objects and environments, enhances the robustness and accuracy of action detection, and achieves a balance between the speed and accuracy of action detection, thereby improving the reliability and efficiency of safety detection.
Smart Images
Figure CN119559558A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular, to a security detection method, device, medium, electronic device and program product. Background Art
[0002] Video-based motion detection has made great progress. In related technologies, most motion detection models combine object recognition and scene analysis to perform motion detection. For example, if you want to detect the action of a robot carrying object a in an assembly line, most existing motion detection models will identify the objects in the video and the scenes in the video, and use this to detect whether the robot on the assembly line has performed the action of carrying object a. However, since most existing motion detection models rely on specific objects and specific environments in the training data, they lack robustness in generalizing to unseen objects and environments, which greatly reduces the effectiveness of motion detection, and thus reduces the reliability and efficiency of safety detection. Summary of the invention
[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.
[0004] In a first aspect, the present disclosure provides a security detection method, comprising: Perform posture detection on each frame in the video to be detected, and use the frame containing the target action of the target object determined by the posture detection result as a key frame; The key frame is processed based on the machine learning model to obtain a frame containing the target action of the target object determined by the machine learning model.
[0005] In a second aspect, the present disclosure provides a safety detection device, comprising: A posture detection module is used to perform posture detection on each frame in the video to be detected, and use the frame containing the target action of the target object determined by the posture detection result as a key frame; A key frame analysis module is used to process the key frame based on a machine learning model to obtain a frame containing the target action of the target object determined by the machine learning model.
[0006] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of any one of the methods described in the first aspect.
[0007] In a fourth aspect, the present disclosure provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of any one of the methods in the first aspect.
[0008] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods in the first aspect.
[0009] By adopting the above technical solution, it is possible to perform posture detection on each frame in the video to be detected, take the frame containing the target action of the target object determined by the posture detection result as a key frame, and process the key frame based on the machine learning model to obtain the frame containing the target action of the target object determined by the machine learning model. Therefore, on the one hand, action detection can be achieved without scene analysis and object detection, which improves the generalization ability of unseen objects and environments and improves the robustness of action detection; on the other hand, through posture detection, most of the frames that obviously do not contain the target action of the target object can be quickly filtered out, and the frames that are most likely to contain the target action of the target object are retained as key frames, and by processing the key frames based on the machine learning model, it is possible to accurately determine whether the key frames that may contain the target action of the target object do contain the target action of the target object, that is, the posture detection results are reviewed, so that the speed and accuracy of action detection are balanced through the two-level action detection processing of posture detection and processing based on the machine learning model, the speed and accuracy of action detection are balanced, the speed and accuracy of action detection are ensured, the effectiveness and confidence of action detection are greatly improved, and the reliability and efficiency of security detection are improved.
[0010] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 It is a flow chart of a security detection method according to an embodiment of the present disclosure.
[0012] Figure 2 This is another flow chart of a security detection method according to an embodiment of the present disclosure.
[0013] Figure 3 It is a schematic block diagram of a safety detection device according to an embodiment of the present disclosure.
[0014] Figure 4 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0016] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0017] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0018] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0019] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0021] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.
[0023] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0025] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.
[0026] In the related art, the video-based motion detection models all combine object recognition and scene analysis to perform motion detection. For example, if you want to detect the action of a robot carrying object a in the assembly line work, most of the existing motion detection models will identify the objects in the video and the scenes in the video, and detect whether the robot on the assembly line has performed the action of carrying object a. These motion detection models all rely on specific objects and specific environments in the training data. Therefore, the scene analysis ability and object recognition ability of the existing motion detection models are limited. For example, most motion detection models often cannot detect objects such as water bottles with different designs. More specifically, when objects or unfamiliar scenes that the motion detection model is not familiar with appear in the video to be detected, no motion detection model can accurately perform motion detection with confidence. Therefore, the generalization of existing motion detection models to unseen objects and unseen environments lacks robustness, which greatly reduces the effectiveness and confidence of motion detection, and thus reduces the reliability and efficiency of safety detection.
[0027] Figure 1 1 is a flow chart of a security detection method according to an embodiment of the present disclosure. The security detection method can be applied to various security detection scenarios, such as detection of assembly line robots or IT room detection, and can be used to detect whether a certain action in a certain scene occurs in a video. Figure 1 As shown, the safety detection method may include the following steps S11 and S12.
[0028] In step S11, posture detection is performed on each frame in the video to be detected, and the frame containing the target action of the target object determined by the posture detection result is used as a key frame.
[0029] Posture detection is a coarse-grained action detection method. Through posture detection, frames that obviously do not contain the target action of the target object can be filtered out from the video to be detected, and frames that do contain the target action of the target object in the video to be detected and frames that are difficult to distinguish whether they contain the target action of the target object are retained as key frames. In this way, through posture detection, frames in which the target action may occur in the video to be detected can be extracted, and unnecessary video clips that obviously do not contain the target action in the video to be detected can be deleted.
[0030] Various posture detection algorithms may be used to perform posture detection, and the present disclosure does not limit this.
[0031] The target object may be, for example, a person, an animal, a machine, a robot on an assembly line, etc. The target action may be various actions that the target object can perform.
[0032] In step S12, the key frame is processed based on the machine learning model to obtain a frame containing a target action of the target object determined by the machine learning model.
[0033] In some embodiments, the machine learning model may be a multimodal large language model, and image analysis may be performed on key frames based on the multimodal large language model to obtain frames containing target actions of the target object determined by the multimodal large language model. The multimodal large language model may be, for example, ChatGPT or other multimodal large language models.
[0034] In some embodiments, hints on how to determine key frames may be input into a multimodal large language model so that the multimodal large language model may consider these hints when performing image analysis to achieve more accurate action detection.
[0035] By adopting the above technical solution, it is possible to perform posture detection on each frame in the video to be detected, take the frame containing the target action of the target object determined by the posture detection result as a key frame, and process the key frame based on the machine learning model to obtain the frame containing the target action of the target object determined by the machine learning model. Therefore, on the one hand, action detection can be achieved without scene analysis and object detection, which improves the generalization ability of unseen objects and environments and improves the robustness of action detection; on the other hand, through posture detection, most of the frames that obviously do not contain the target action of the target object can be quickly filtered out, and the frames that are most likely to contain the target action of the target object are retained as key frames, and by processing the key frames based on the machine learning model, it is possible to accurately determine whether the key frames that may contain the target action of the target object do contain the target action of the target object, that is, the posture detection results are reviewed, so that the speed and accuracy of action detection are balanced through the two-level action detection processing of posture detection and processing based on the machine learning model, the speed and accuracy of action detection are balanced, the speed and accuracy of action detection are ensured, the effectiveness and confidence of action detection are greatly improved, and the reliability and efficiency of security detection are improved.
[0036] In some embodiments, the step S11 of performing posture detection on each frame in the video to be detected and taking the frame containing the target action of the target object determined by the posture detection result as the key frame may include: performing posture detection on each frame in the video to be detected, and if it is determined that in a preset number of consecutive frames, the distances between the parts of the target object involved in making the target action all meet the preset distance conditions, then taking the preset number of consecutive frames as the key frames containing the target action of the target object determined by the posture detection result.
[0037] Take the robot carrying object a in the assembly line as an example. First, the positions of the various parts that the robot needs to use to carry object a in each frame are determined through posture detection. Then, if the distances between these parts of the robot in a preset number of consecutive frames all meet the preset distance conditions, such as being lower than the preset distance threshold, then these preset number of consecutive frames are marked as key frames.
[0038] In some embodiments, the step S11 of performing posture detection on each frame in the video to be detected and using the frame containing the target action of the target object determined by the posture detection result as a key frame may include: performing posture detection on each frame in the video to be detected, and if it is determined that the distances between the parts involved in the target action of the target object in a preset number of consecutive frames do not all meet the preset distance condition, then determining that the preset number of consecutive frames are not key frames containing the target action of the target object determined by the posture detection result. For example, among the preset number of consecutive frames, there is one or more frames in which the distances between the parts involved in the target action of the target object do not meet the preset distance condition, then the preset number of consecutive frames are not key frames.
[0039] The preset number of consecutive frames can be dynamically determined according to the frame rate of the video to be detected. For example, the preset number of consecutive frames can be set to a preset percentage of the frame rate of the video to be detected. In this way, the preset number of consecutive frames can be changed according to the frame rate of the video to be detected, so that the security detection method according to the embodiment of the present disclosure can adapt to videos to be detected with different frame rates.
[0040] The preset distance threshold in the preset distance condition can be dynamically determined according to the distance between the parts of the target object in the video to be detected whose relative positions are fixed, for example, dynamically determined according to the preset percentage of the distance between the parts of the target object in the video to be detected whose relative positions are fixed. For example, if the target object is a robot in an assembly line, the parts with fixed relative positions may include joints of the robot with fixed relative positions. Usually, when the target object is close to the camera, the target object will occupy more pixels in the frame of the video to be detected, thereby increasing the pixel distance between the parts of the target object. When the target object is far away from the camera, the target object will occupy fewer pixels in the frame of the video to be detected, thereby reducing the pixel distance between the parts of the target object. Therefore, by dynamically determining the preset distance threshold in the preset distance condition according to the distance between the parts of the target object in the video to be detected whose relative positions are fixed, it can be ensured that the preset distance threshold is dynamically adjusted according to the distance between the target object and the camera, solving the perspective distance threshold problem.
[0041] By adopting the above technical solution, it is possible to efficiently determine the key frames of the target action that may contain the target object through posture detection.
[0042] Figure 2 This is another flow chart of a security detection method according to an embodiment of the present disclosure.
[0043] like Figure 2 As shown, first, in step S21, a video to be detected is obtained.
[0044] Then, in step S22, it is determined whether a frame is extracted from the video to be detected. If a frame is extracted, it means that the video to be detected has not been processed, and then the process turns to step S23. If no frame is extracted, it means that the video to be detected has been processed, and then the process turns to step S28 to end the security detection process.
[0045] In step S23, gesture detection is performed on the extracted frame. Various gesture detection algorithms may be used to perform gesture detection, and the present disclosure does not limit this.
[0046] In step S24, based on the posture detection result, it is determined whether the distance between the parts of the target object involved in the target action meets the preset distance condition. The method for determining the preset distance threshold has been described in detail above and will not be repeated here.
[0047] If the distance between the parts of the target object involved in the target action meets the preset distance condition, the process goes to step S25. If the distance between the parts of the target object involved in the target action does not meet the preset distance condition, the process goes back to step S22.
[0048] In step S25, it is determined whether the preset number of consecutive frames has been reached. If the preset number of consecutive frames has not been reached, the process returns to step S22. If the preset number of consecutive frames has been reached, the process returns to step S22 to continue the posture detection of the next frame, and turns to step S26 to determine the key frame. The method for determining the preset number of consecutive frames has been described in detail above and will not be repeated here.
[0049] In step S26, a key frame is determined. The implementation method of determining the key frame has been described in detail above and will not be repeated here.
[0050] In step S27, the key frame is processed based on the machine learning model to obtain a frame containing the target action of the target object determined by the machine learning model. The implementation method of processing the key frame has been described in detail above and will not be repeated here.
[0051] By adopting the above technical solution, it is possible to perform posture detection on each frame in the video to be detected, take the frame containing the target action of the target object determined by the posture detection result as a key frame, and process the key frame based on the machine learning model to obtain the frame containing the target action of the target object determined by the machine learning model. Therefore, on the one hand, action detection can be achieved without scene analysis and object detection, which improves the generalization ability of unseen objects and environments and improves the robustness of action detection; on the other hand, through posture detection, most of the frames that obviously do not contain the target action of the target object can be quickly filtered out, and the frames that are most likely to contain the target action of the target object are retained as key frames, and by processing the key frames based on the machine learning model, it is possible to accurately determine whether the key frames that may contain the target action of the target object do contain the target action of the target object, that is, the posture detection results are reviewed, so that the speed and accuracy of action detection are balanced through the two-level action detection processing of posture detection and processing based on the machine learning model, the speed and accuracy of action detection are balanced, the speed and accuracy of action detection are ensured, the effectiveness and confidence of action detection are greatly improved, and the reliability and efficiency of security detection are improved.
[0052] Figure 3 is a schematic block diagram of a safety detection device according to an embodiment of the present disclosure. The safety detection device can be applied to various safety detection scenarios, such as detection of assembly line robots or IT room detection, and can be used to detect whether a certain action in a certain scene occurs in a video. Figure 3 As shown, the security detection device 30 includes: a posture detection module 31, which is used to perform posture detection on each frame in the video to be detected, and use the frame containing the target action of the target object determined by the posture detection result as the key frame; a key frame analysis module 32, which is used to process the key frame based on the machine learning model, and obtain the frame containing the target action of the target object determined by the machine learning model.
[0053] By adopting the above technical solution, it is possible to perform posture detection on each frame in the video to be detected, take the frame containing the target action of the target object determined by the posture detection result as a key frame, and process the key frame based on the machine learning model to obtain the frame containing the target action of the target object determined by the machine learning model. Therefore, on the one hand, action detection can be achieved without scene analysis and object detection, which improves the generalization ability of unseen objects and environments and improves the robustness of action detection; on the other hand, through posture detection, most of the frames that obviously do not contain the target action of the target object can be quickly filtered out, and the frames that are most likely to contain the target action of the target object are retained as key frames, and by processing the key frames based on the machine learning model, it is possible to accurately determine whether the key frames that may contain the target action of the target object do contain the target action of the target object, that is, the posture detection results are reviewed, so that the speed and accuracy of action detection are balanced through the two-level action detection processing of posture detection and processing based on the machine learning model, the speed and accuracy of action detection are balanced, the speed and accuracy of action detection are ensured, the effectiveness and confidence of action detection are greatly improved, and the reliability and efficiency of security detection are improved.
[0054] Optionally, the posture detection module 31 performs posture detection on each frame in the video to be detected, and uses the frame containing the target action of the target object determined by the posture detection result as a key frame, including: Perform posture detection on each frame in the video to be detected. If it is determined that in a preset number of consecutive frames, the distances between the parts of the target object involved in making the target action all meet the preset distance conditions, then the preset number of consecutive frames are used as key frames containing the target action of the target object determined by the posture detection results.
[0055] Optionally, the safety detection device 30 may further include a first determination module, configured to: The preset number of the continuous frames is dynamically determined according to the frame rate of the video to be detected.
[0056] Optionally, the safety detection device 30 may further include a second determination module, configured to: The preset distance threshold in the preset distance condition is dynamically determined according to the distance between the parts of the target object in the to-be-detected video whose relative positions are fixed.
[0057] Optionally, the posture detection module 31 performs posture detection on each frame in the video to be detected, and uses the frame containing the target action of the target object determined by the posture detection result as a key frame, including: Perform posture detection on each frame in the video to be detected. If it is determined that in a preset number of consecutive frames, the distances between the parts of the target object involved in performing the target action do not all meet the preset distance conditions, then it is determined that the preset number of consecutive frames are not key frames containing the target action of the target object determined by the posture detection results.
[0058] Optionally, the machine learning model is a multimodal large language model, and the key frame analysis module 32 processes the key frames based on the machine learning model, including: Image analysis is performed on the key frames based on a multimodal large language model.
[0059] The specific implementation of the operations performed by each module in the security detection device according to the embodiment of the present disclosure has been described in detail in the relevant security detection method, and will not be repeated here.
[0060] The present disclosure also provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processing device, the steps of any method in the present disclosure are implemented.
[0061] The present disclosure also provides an electronic device, comprising: a storage device having a computer program stored thereon; A processing device is used to execute a computer program in a storage device to implement the steps of any method in the present disclosure.
[0062] The present disclosure also provides a computer program product, comprising a computer program, which implements the steps of any method in the present disclosure when executed by a processor.
[0063] Reference below Figure 4 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0064] like Figure 4As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0065] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0066] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0067] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0068] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0069] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0070] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: performs posture detection on each frame in the video to be detected, and takes the frame containing the target action of the target object determined by the posture detection result as the key frame; processes the key frame based on the machine learning model, and obtains the frame containing the target action of the target object determined by the machine learning model.
[0071] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0072] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0073] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the posture detection module may also be described as "a module that performs posture detection on each frame in the video to be detected, and uses the frame containing the target action of the target object determined by the posture detection result as a key frame".
[0074] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0075] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0076] According to one or more embodiments of the present disclosure, Example 1 provides a security detection method, including: Perform posture detection on each frame in the video to be detected, and use the frame containing the target action of the target object determined by the posture detection result as a key frame; The key frame is processed based on the machine learning model to obtain a frame containing the target action of the target object determined by the machine learning model.
[0077] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein the step of performing posture detection on each frame in the video to be detected and taking the frame containing the target action of the target object determined by the posture detection result as a key frame includes: Perform posture detection on each frame in the video to be detected. If it is determined that in a preset number of consecutive frames, the distances between the parts of the target object involved in performing the target action all meet the preset distance conditions, then the preset number of consecutive frames are used as key frames containing the target action of the target object determined by the posture detection results.
[0078] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 2, wherein the method further includes: The preset number of the continuous frames is dynamically determined according to the frame rate of the video to be detected.
[0079] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 2, wherein the method further includes: The preset distance threshold in the preset distance condition is dynamically determined according to the distance between the parts of the target object in the to-be-detected video whose relative positions are fixed.
[0080] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 1, wherein the step of performing posture detection on each frame in the video to be detected and taking the frame containing the target action of the target object determined by the posture detection result as a key frame includes: Perform posture detection on each frame in the video to be detected. If it is determined that in a preset number of consecutive frames, the distances between the parts of the target object involved in performing the target action do not all meet the preset distance conditions, then it is determined that the preset number of consecutive frames are not key frames containing the target action of the target object determined by the posture detection results.
[0081] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 1, wherein the machine learning model is a multimodal large language model, and the processing of the key frame based on the machine learning model includes: Image analysis is performed on the key frames based on a multimodal large language model.
[0082] According to one or more embodiments of the present disclosure, Example 7 provides a safety detection device, including: A posture detection module is used to perform posture detection on each frame in the video to be detected, and use the frame containing the target action of the target object determined by the posture detection result as a key frame; A key frame analysis module is used to process the key frame based on a machine learning model to obtain a frame containing the target action of the target object determined by the machine learning model.
[0083] According to one or more embodiments of the present disclosure, Example 8 provides a computer-readable medium having a computer program stored thereon, which implements the steps of any of the methods described in Examples 1-6 when executed by a processing device.
[0084] According to one or more embodiments of the present disclosure, Example 9 provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of any one of the methods described in Examples 1-6.
[0085] According to one or more embodiments of the present disclosure, Example 10 provides a computer program product, including a computer program, which implements the steps of any one of the methods of Examples 1 to 6 when executed by a processor.
[0086] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0087] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0088] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Claims
1. A safety detection method, characterized in that: include: Perform posture detection on each frame in the video to be detected, and use the frame containing the target action of the target object determined by the posture detection result as a key frame; The key frame is processed based on the machine learning model to obtain a frame containing the target action of the target object determined by the machine learning model.
2. The method according to claim 1, characterized in that The method of performing posture detection on each frame in the video to be detected and taking the frame containing the target action of the target object determined by the posture detection result as a key frame includes: Perform posture detection on each frame in the video to be detected. If it is determined that in a preset number of consecutive frames, the distances between the parts of the target object involved in performing the target action all meet a preset distance condition, then the preset number of consecutive frames are used as key frames containing the target action of the target object determined by the posture detection results.
3. The method according to claim 2, characterized in that The method further comprises: The preset number of the continuous frames is dynamically determined according to the frame rate of the video to be detected.
4. The method according to claim 2, characterized in that: The method further comprises: The preset distance threshold in the preset distance condition is dynamically determined according to the distance between the parts of the target object in the to-be-detected video whose relative positions are fixed.
5. The method according to claim 1, characterized in that The method of performing posture detection on each frame in the video to be detected and taking the frame containing the target action of the target object determined by the posture detection result as a key frame includes: Perform posture detection on each frame in the video to be detected. If it is determined that in a preset number of consecutive frames, the distances between the parts of the target object involved in performing the target action do not all meet the preset distance conditions, then it is determined that the preset number of consecutive frames are not key frames containing the target action of the target object determined by the posture detection results.
6. The method according to claim 1, characterized in that The machine learning model is a multimodal large language model, and the processing of the key frame based on the machine learning model includes: Perform image analysis on the key frames based on the multimodal large language model.
7. A safety detection device, characterized in that: include: A posture detection module is used to perform posture detection on each frame in the video to be detected, and use the frame containing the target action of the target object determined by the posture detection result as a key frame; A key frame analysis module is used to process the key frame based on a machine learning model to obtain a frame containing the target action of the target object determined by the machine learning model.
8. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 6 are implemented.
9. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.