Method and device for identifying target object behavior

Through the linkage between panoramic cameras and details cameras, the behavior of target objects is identified using pixel position mapping relationships, which solves the problem of poor clarity and accuracy in large-scale applications of conventional cameras, and achieves higher behavior recognition accuracy.

CN118781525BActive Publication Date: 2024-11-26ZHEJIANG DAHUA TECH CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411256986.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-09
Publication Date
2024-11-26
Estimated Expiration
2044-09-09

AI Technical Summary

Technical Problem

In application scenarios with a large range, the images obtained by conventional panoramic cameras that are far away from the camera may have poor clarity and too small targets, which seriously affects the accuracy of the behavioral analysis algorithm.

Method used

The panoramic video frame and sub-region video frame are collected separately in the same time range through the panoramic camera and the detail camera, and the behavior of the target object is identified using the preset pixel position mapping relationship, removing the behavior of non-target objects, and improving the accuracy of behavior recognition.

Benefits of technology

It effectively solves the problem of poor behavior recognition effect and improves the accuracy of behavior recognition. Especially when the target area is large, it improves the clarity and recognition accuracy of the target object.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118781525B_ABST
    Figure CN118781525B_ABST
Patent Text Reader

Abstract

The present invention provides a method and device for identifying the behavior of a target object, including: obtaining a plurality of video frames obtained by shooting a target area by a panoramic camera within a target time range, obtaining a first video frame set, wherein the shooting angle of the panoramic camera covers the target area; obtaining a plurality of video frames obtained by shooting N sub-areas included in the target area by a detail camera at N preset points within the target time range, obtaining a second video frame set, wherein N is an integer greater than 1, and the shooting angle of the detail camera at a preset point covers a sub-area; identifying the behavior of each target object in the target area through the first video frame set, the second video frame set, and a preset pixel position mapping relationship. Through the present invention, the problem of poor behavior recognition effect in the related art is solved, thereby achieving the effect of improving the accuracy of behavior recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence, and in particular, to a method and device for identifying target object behavior. Background Art

[0002] Behavior analysis algorithms are a technology that infers the intentions and goals of individuals or groups by identifying and analyzing their behavior patterns and trends. This algorithm can be used to analyze and count the behavior of target objects, thereby helping users better predict behavior and manage it accordingly, such as online test behavior, consumption behavior, and traffic behavior.

[0003] However, in actual applications, for application scenarios with a larger range, the images acquired by conventional panoramic cameras may have problems such as poor clarity and small size for targets far away from the camera, which will seriously affect the accuracy of algorithm analysis.

[0004] There is currently no effective solution to the above problems. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for identifying the behavior of a target object, so as to at least solve the problem of poor behavior recognition effect in the related art.

[0006] According to an embodiment of the present invention, a method for identifying the behavior of a target object is provided, comprising: acquiring a plurality of video frames obtained by photographing a target area by a panoramic camera within a target time range to obtain a first video frame set, wherein the shooting angle of the panoramic camera covers the target area; acquiring a plurality of video frames obtained by photographing N sub-areas included in the target area by a detail camera at N preset points within the target time range to obtain a second video frame set, wherein N is an integer greater than 1, and the shooting angle of the detail camera at one of the preset points covers one of the sub-areas; identifying the behavior of each target object in the target area through the first video frame set, the second video frame set, and a preset pixel position mapping relationship, wherein the pixel position mapping relationship records a mapping relationship between pixel positions in the video frames photographed by the panoramic camera and pixel positions in the video frames photographed by the detail camera at each of the preset points, the target object is an object other than a first object appearing in the target area, and the first object satisfies at least one of the following: a difference between a moving distance of the target object and a moving distance of the first object is greater than a first preset threshold, and a difference between an imaging area of ​​the first object and an imaging area of ​​the target object is greater than a second preset threshold.

[0007] In an exemplary embodiment, the behavior of each target object in the target area is identified through the first video frame set, the second video frame set, and a preset pixel position mapping relationship, including: inputting each video frame included in the first video frame set into the first target neural network model in sequence, and identifying the first object in the target area through the first target neural network model; inputting each video frame included in the second video frame set into the second target neural network model in sequence, and identifying the objects in the target area and the behavior of each object through the second target neural network model; removing the behavior of the first object from the behaviors of each object identified by the second target neural network model through the preset pixel position mapping relationship to obtain the behavior of the target object.

[0008] In an exemplary embodiment, the behavior of the target object is obtained by removing the behavior of the first object from the behaviors of the various objects recognized by the second target neural network model through the preset pixel position mapping relationship, including: when the i-th video frame included in the second video frame set is a video frame shot by the detail camera at the k-th preset point at the j-th time point, the i-th video frame is input into the second target neural network model, and each object in the i-th video frame is recognized by the second target neural network model to obtain a plurality of detection frames of the object, wherein i, j, and k are integers; in the first video frame set, the video frame shot by the panoramic camera at the j-th time point is determined to obtain the S-th video frame, wherein S is an integer; the first detection frame obtained by the first target neural network model for recognizing the first object in the S-th video frame; and the behavior of the target object is determined by the plurality of detection frames, the first detection frame, and the pixel position mapping relationship obtained by the second target neural network model for recognizing the i-th video frame.

[0009] In an exemplary embodiment, the behavior of the target object is determined by using the second target neural network model to recognize the i-th video frame using the multiple detection frames, the first detection frame, and the preset pixel position mapping relationship, including: determining a second detection frame that matches the first detection frame among the multiple detection frames according to the pixel position mapping relationship; determining objects within detection frames other than the second detection frame among the multiple detection frames recognized by the second target neural network model for the i-th video frame as the target object; and obtaining the behavior recognition result of the target object in the i-th video frame using the second target neural network model.

[0010] In an exemplary embodiment, a second detection frame matching the first detection frame is determined in multiple detection frames according to the pixel position mapping relationship, including: mapping the first detection frame in the Sth video frame to the i-th video frame according to the pixel position mapping relationship to obtain a first mapped detection frame; and determining the detection frame with the largest overlap rate between the multiple detection frames in the i-th video frame and the first mapped detection frame as the second detection frame.

[0011] In an exemplary embodiment, each video frame of the first video frame set is sequentially input into a first target neural network model, and the first object in the target area is identified by the first target neural network model, including: analyzing the movement trajectory of each object in the first video frame set by a trajectory analysis module of the first target neural network model, and determining the object whose difference in movement distance with other objects in the first video frame set is greater than or equal to the first preset threshold as the first target object; identifying the identity of the first target object by an identity recognition module of the first target neural network model to obtain an identity tag of the first target moving object; in the case where the identity tag of the first target moving object is a target identity tag, determining the first target moving object as the first object; or The imaging area analysis module of the first target neural network model analyzes the imaging areas of each object in the first video frame set, and determines the object whose difference with the imaging area of ​​other objects in the first video frame set is greater than or equal to the second preset threshold as the second target object; the identity of the second target moving object is identified by the identity recognition module of the first target neural network model to obtain the identity tag of the second target moving object; when the identity tag of the second target moving object is the target identity tag, the second target moving object is determined to be the first object; after the first object in the target area is identified by the first target neural network model, the method also includes: displaying the position information of the first object in real time in the video frame obtained by shooting the target area with the panoramic camera.

[0012] In an exemplary embodiment, after obtaining the behavior recognition result of the target object in the i-th video frame by the second target neural network model, the method further includes at least one of the following: displaying the behavior recognition result of each target object on a target display device; when the behavior recognition result of each target object indicates the presence of abnormal behavior, determining the target position of the target object with the abnormal behavior in the target area based on the detection frame obtained by the second target neural network model for recognizing the target object, and sending a prompt message to the terminal device at the target position.

[0013] According to another embodiment of the present invention, there is provided a device for identifying the behavior of a target object, comprising: a first acquisition module, for acquiring a plurality of video frames obtained by photographing a target area by a panoramic camera within a target time range, to obtain a first video frame set, wherein the shooting angle of the panoramic camera covers the target area; a second acquisition module, for acquiring a plurality of video frames obtained by photographing N sub-areas included in the target area by a detail camera at N preset points within the target time range, to obtain a second video frame set, wherein N is an integer greater than 1, and the shooting angle of the detail camera at one of the preset points covers one of the sub-areas; an identification module, for obtaining a second video frame set by means of the The first video frame set, the second video frame set, and a preset pixel position mapping relationship identify the behavior of each target object in the target area, wherein the pixel position mapping relationship records the mapping relationship between the pixel positions in the video frames taken by the panoramic camera and the pixel positions in the video frames taken by the detail camera at each of the preset points, and the target object is an object other than the first object that appears in the target area, and the first object satisfies at least one of the following: the difference between the moving distance of the target object and the moving distance of the first object is greater than a first preset threshold, and the difference between the imaging area of ​​the first object and the imaging area of ​​the target object is greater than a second preset threshold.

[0014] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0015] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0016] According to yet another embodiment of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0017] Through the present invention, since the panoramic camera and the detail camera respectively collect panoramic video frames and sub-area video frames within the same time range, and the behavior of the target object is recognized according to the mapping relationship between the video frames collected by the two cameras, the problem of poor behavior recognition effect in the related art can be solved, and the accuracy of behavior recognition can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1is a hardware structure block diagram of a mobile terminal according to a method for identifying target object behavior according to an embodiment of the present invention;

[0019] Figure 2 is a flow chart of a method for identifying target object behavior according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of a video frame captured by a panoramic camera according to an embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of a video frame captured by a detail camera according to an embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of a mapping relationship according to an embodiment of the present invention;

[0023] Figure 6 is a flow chart of behavior recognition according to an embodiment of the present invention;

[0024] Figure 7 is a schematic diagram of a layout scene according to an embodiment of the present invention;

[0025] Figure 8 4 is a structural block diagram of an apparatus for identifying target object behavior according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal for a method of identifying target object behavior according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0029] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for identifying the behavior of the target object in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] In this embodiment, a method running on the above mobile terminal is provided. Figure 2 is a flow chart of a method for identifying target object behavior according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0032] Step S202, acquiring a plurality of video frames obtained by shooting a target area by a panoramic camera within a target time range to obtain a first video frame set, wherein the shooting angle of the panoramic camera covers the target area;

[0033] The panoramic camera can be a device for taking pictures or videos, such as a gun camera, a zoom camera, etc., and all objects in the target area can be photographed by the panoramic camera; the target time range can be any time period, such as during an exam, during class, or one hour, two hours, etc. Optionally, during class, a panoramic camera can be used to photograph the situation in a classroom to obtain a first video frame set, which includes multiple video frames taken by the panoramic camera during class, wherein the content in any video frame can be as follows: Figure 3 shown.

[0034] Step S204, obtaining a plurality of video frames obtained by respectively photographing N sub-areas included in the target area at N preset points by the detail camera within the target time range, to obtain a second video frame set, wherein N is an integer greater than 1, and the shooting angle of view of the detail camera at one of the preset points covers one of the sub-areas;

[0035] The above-mentioned detail camera can be a device for taking pictures or videos, such as a ball camera, a pan-tilt camera, etc. The detail camera can be used to capture objects in a part of the target area; the above-mentioned target time range can be any time period, such as during an exam, during class, or a certain 1 hour, two hours, etc.; the above-mentioned preset point can be the shooting point of the detail camera, and shooting at any preset point can obtain a video frame of the sub-area corresponding to the preset point. Optionally, the classroom is divided into multiple area blocks according to the size of the classroom, for example: 2x4, 3x6 and other size area blocks, the detail camera sets a preset point according to each image block, and the camera adjusts the angle and magnification of each preset point according to the distance of the image block, so that the number of people in each area block is maintained at about 8-10 people, and the human body width pixels are at least greater than 100 pixels. While students are in class, the situation in a classroom is captured by a detail camera to obtain a second video frame set, which includes multiple video frames captured by the detail camera at each preset point during class, wherein the content in any video frame can be as follows Figure 4 shown.

[0036] Step S206, identifying the behavior of each target object in the target area through the first video frame set, the second video frame set, and a preset pixel position mapping relationship, wherein the pixel position mapping relationship records the mapping relationship between the pixel positions in the video frames taken by the panoramic camera and the pixel positions in the video frames taken by the detail camera at each of the preset points, and the target object is an object other than the first object that appears in the target area, and the first object satisfies at least one of the following: the difference between the moving distance of the target object and the moving distance of the first object is greater than a first preset threshold, and the difference between the imaging area of ​​the first object and the imaging area of ​​the target object is greater than a second preset threshold.

[0037] The above mapping relationship can be the pixel position corresponding to the pixel in the video frame captured by the detail camera in the video frame captured by the panoramic camera (such as Figure 5), wherein each video frame can be composed of multiple pixel blocks, and the panoramic camera and the detail camera are spatially calibrated by the panoramic video frames of the target area collected within the target time range and the mapping relationship between each pixel block. The calibration method can be a multi-point calibration method, a multi-camera calibration method based on Kalibr, etc.; the target object can be an object recognized by behavior, such as a student during an exam, or a customer in a convenience store, etc.; the first object can be an object that does not need to be recognized by behavior, such as an invigilator during an exam, or a cashier in a convenience store, etc.; the first object can be judged by the difference between it and the target object, such as movement displacement, clothing, worn marks, facial features, etc.; optionally, in an exam scenario, since the invigilator needs to walk back and forth to supervise the exam, and the students just sit and answer questions, the movement displacement Determine whether it is an invigilator, for example, determine the object whose moving distance difference with the target object (examinee) in the target area within the target time range exceeds the first preset threshold as the invigilator (first object), so that when there is no face image on the detail camera, the identity of the teacher can also be determined, so as to achieve the purpose of only analyzing the student behavior, or, in a convenience store, customers will walk back and forth to select goods, but the cashier will only help the customers who have selected the goods to check out at the cashier counter, so it can be determined whether it is a cashier by moving displacement, for example, determine the object whose moving distance difference with the target object (cashier) in the target area within the target time range is greater than the first preset threshold as the cashier (first object), so that when there is no face image on the detail camera (the cashier bows his head to check out), the identity of the cashier can also be determined, so as to achieve the purpose of only analyzing the customer behavior. At the same time, the behavior of each target object in the target area is identified through the first video frame set, the second video frame set, and the preset pixel position mapping relationship.

[0038] Optionally, the first object can also be determined by the size of the imaging area of ​​the first object and the target object. For example, in an examination scenario, since the invigilator needs to walk back and forth to supervise the examination, while the students just sit and answer questions, the object whose imaging area difference between the target object (examinee) and the target object (examinee) within the target area within the target time range exceeds the second preset threshold can be determined as the invigilator (first object). Alternatively, in a convenience store, customers will walk back and forth to select goods, and the camera can capture the whole body, but the cashier only stands at the cashier counter to help customers who have selected goods check out, and the camera can only capture the upper body. Therefore, the object whose imaging area difference between the target object (cashier) and the target object (cashier) within the target area within the target time range is greater than the second preset threshold is determined as the cashier (first object), thereby achieving the purpose of determining the first object even when there is no face image on the detail camera.

[0039] Specifically, each video frame included in the first video frame set is sequentially input into the first target neural network model, and the first object in the target area is identified by the first target neural network model; each video frame included in the second video frame set is sequentially input into the second target neural network model, and the objects in the target area and the behavior of each object are identified by the second target neural network model; the behavior of the first object is removed from the behaviors of each object identified by the second target neural network model through the preset pixel position mapping relationship to obtain the behavior of the target object.

[0040] The above-mentioned first target neural network model is used to identify the first object in the target area: first, through target detection technology, such as YOLO, SSD, FastRcnn, etc., multiple labels are detected, such as face, head, head and shoulders, upper body, whole body, etc., to identify the human target in the target area; then, based on the target tracking information technology, the motion trajectory of the human target is identified to determine whether the human target has moving behavior; finally, through comprehensive judgment of the moving behavior of the human target, face recognition, whole body visible information, etc., the first object in the target area is identified.

[0041] The above-mentioned second target neural network model can be used to identify the behavior of target objects in the target area: first, through target detection technology, such as YOLO, SSD, FastRcnn, etc., multiple labels are detected, such as face, head, head and shoulders, upper body, whole body, etc., to identify human targets in the target area; then, behavior recognition is realized based on key point + posture classification, etc., to obtain the behavior of each human target, and realize the identification of the behavior of the target object in the target area.

[0042] At the same time, through the mapping relationship, the first object in each video frame in the second video frame set is excluded to achieve behavior recognition of the target object.

[0043] Optionally, the executor of the above steps may be a background processor, or other devices with similar processing capabilities, or a machine that integrates at least an image acquisition device and a data processing device, wherein the image acquisition device may include a graphics acquisition module such as a camera, and the data processing device may include a computer, a mobile phone or other terminal, but is not limited thereto.

[0044] Through the above steps, through the multi-camera linkage mode, the identity of the first object is determined when there is no face image on the detail camera, so as to achieve the purpose of only analyzing the behavior of the target object. At the same time, the use of the detail camera to analyze the behavior of the person, especially when the target area is large, makes each object in the target area have higher clarity, and improves the accuracy of the recognition algorithm. The problem of poor behavior recognition effect in the related technology is solved, and the accuracy of behavior recognition is improved.

[0045] It should be noted that the execution order of step S202 and step S204 is interchangeable, that is, step S104 may be executed first, and then step S102.

[0046] As an optional implementation, the behavior of the target object is obtained by removing the behavior of the first object from the behaviors of the various objects recognized by the second target neural network model through the preset pixel position mapping relationship, including: when the i-th video frame included in the second video frame set is a video frame shot by the detail camera at the k-th preset point at the j-th time point, the i-th video frame is input into the second target neural network model, and each object in the i-th video frame is recognized by the second target neural network model to obtain a plurality of detection frames of the object, wherein i, j, and k are integers; in the first video frame set, the video frame shot by the panoramic camera at the j-th time point is determined to obtain the S-th video frame, wherein S is an integer; the first detection frame obtained by the first target neural network model for recognizing the first object in the S-th video frame; and the behavior of the target object is determined by the plurality of detection frames, the first detection frame, and the pixel position mapping relationship obtained by the second target neural network model for recognizing the i-th video frame.

[0047] A panoramic video frame taken by a panoramic camera and a video frame of a sub-area taken by a detail camera at the same time point are obtained, wherein the sub-area is a sub-area corresponding to a preset point where the detail camera is located at the same time point, and the panoramic video frame and the video frame of the sub-area are respectively input into a first target neural network model and a second target neural network model to identify and determine a first object and the behavior of multiple objects in the video frame of the sub-area; according to the mapping relationship, the first object is determined in the video frame of the sub-area and excluded, so as to obtain only the behavior of the target object.

[0048] As an optional implementation, the behavior of the target object is determined by using the second target neural network model to recognize the i-th video frame using the multiple detection frames, the first detection frame, and the preset pixel position mapping relationship, including: determining a second detection frame that matches the first detection frame among the multiple detection frames according to the pixel position mapping relationship; determining objects within detection frames other than the second detection frame among the multiple detection frames recognized by the second target neural network model for the i-th video frame as the target object; and obtaining the behavior recognition result of the target object in the i-th video frame using the second target neural network model.

[0049] As an optional implementation, determining a second detection frame that matches the first detection frame from among the multiple detection frames according to the pixel position mapping relationship includes: mapping the first detection frame in the Sth video frame to the i-th video frame according to the pixel position mapping relationship to obtain a first mapping detection frame; and determining the detection frame that has the largest overlap rate with the first mapping detection frame among the multiple detection frames in the i-th video frame as the second detection frame.

[0050] Optionally, the detection frames of the multiple objects in the i-th video frame may be mapped to the S-th video frame according to the pixel position mapping relationship to obtain multiple third detection frames, and it may be determined whether there are detection frames overlapping with the first detection frame among the third detection frames. If there are detection frames overlapping with the first detection frame among the multiple third detection frames, objects in the other third detection frames except the first detection frame among the multiple third detection frames are determined as target objects; if there are no detection frames overlapping with the first detection frame among the multiple third detection frames, all objects in the multiple third detection frames are determined as target objects.

[0051] As an optional implementation, each video frame of the first video frame set is sequentially input into a first target neural network model, and the first object in the target area is identified by the first target neural network model, including: analyzing the moving trajectory of each object in the first video frame set by the trajectory analysis module of the first target neural network model, and determining the object whose difference in moving distance with other objects in the first video frame set is greater than or equal to the first preset threshold as the first target object; identifying the identity of the first target object by the identity recognition module of the first target neural network model to obtain the identity tag of the first target moving object; in the case where the identity tag of the first target moving object is the target identity tag, determining the first target moving object as the first object; or, analyzing the imaging area of ​​each object in the first video frame set by the imaging area analysis module of the first target neural network model, and determining the object whose difference in imaging area with other objects in the first video frame set is greater than or equal to the second preset threshold as the second target object; identifying the identity of the second target moving object by the identity recognition module of the first target neural network model to obtain the identity tag of the second target moving object; in the case where the identity tag of the second target moving object is the target identity tag, determining the second target moving object as the first object.

[0052] For example, in an examination scenario, since the invigilator needs to walk back and forth to supervise the exam, while the students just sit and answer questions, the object can be determined by the movement displacement. For example, an object whose difference with the target object (examinee) within the target area within the target time range is greater than or equal to a first preset threshold is determined as the invigilator (first object), or an object whose difference with the imaging area of ​​the target object (examinee) within the target area within the target time range exceeds a second preset threshold is determined as the invigilator (first object).

[0053] It should be noted that the above-mentioned mobile objects are only for describing the present embodiment in detail. The moving trajectories of various objects may also be analyzed according to actual conditions to obtain fixed objects, wherein the mobile objects are objects without moving trajectories, etc., which are not limited here.

[0054] As an optional implementation, after identifying the first object in the target area through the first target neural network model, the method further includes: displaying the location information of the first object in real time in the video frame obtained by photographing the target area with the panoramic camera.

[0055] The position information of the first object can be represented by coordinates or the position of the center point of the identification frame of the first object, and displayed in real time through a display device. As an optional embodiment, after obtaining the behavior recognition result of the target object in the i-th video frame by the second target neural network model, the method further includes at least one of the following: displaying the behavior recognition result of each target object on a target display device; when the behavior recognition result of each target object indicates the existence of abnormal behavior, determining the target position of the target object in the target area with the abnormal behavior based on the detection frame obtained by the second target neural network model for identifying the target object, and sending a prompt message to the terminal device at the target position.

[0056] As an optional implementation, the above embodiment may be a method for identifying examinee behavior in an examination scenario. Figure 6 is a flow chart of behavior recognition implemented according to the present invention, such as Figure 6 As shown, the specific process is as follows:

[0057] Step S1, multiple preset point settings: The specific layout of the master and slave cameras can be as follows: Figure 7 As shown, the classroom (target area) is divided into multiple blocks according to its size, such as 2x4, 3x6 and other size blocks. The detail camera sets a preset point according to each image block. The camera adjusts the angle and magnification of each preset point according to the distance of the image block, so that the number of people in each block is kept at about 8-10, and the human body width pixels are at least greater than 100 pixels.

[0058] Step S2, the panoramic camera detects people in real time: Identify the position of the invigilator (first object). After the invigilator moves to the examinee (target object) area, there is no guarantee that there will always be a suitable face image for identity confirmation (for example, when the invigilator has his back or side facing the camera). Therefore, it is necessary to determine whether the face or person target is moving, and whether the whole body is visible at certain moments (the examinee is sitting in the seat during the exam and cannot see the whole body), to confirm whether the person target is the invigilator. The specific implementation method is:

[0059] First, human targets are detected through deep learning target detection technology. Multiple labels can be considered for detection, such as face, head, head and shoulders, upper body, whole body, etc. Common target detection methods include YOLO, SSD / FastRcnn, etc.

[0060] Secondly, based on the target tracking information, the target motion trajectory can be obtained, and by analyzing the target motion trajectory, it can be determined whether the target is moving;

[0061] Finally, through comprehensive judgment of facial recognition, target movement, and target's full body visible information, the identity of the invigilator is confirmed and the invigilator's location is output in real time.

[0062] Step S3, detail camera examinee student behavior: identification behaviors include: writing, lying on the table, turning around, raising hands, etc. The specific implementation method is:

[0063] First, human targets are detected through deep learning target detection technology. Multiple labels can be considered for detection, such as face, head, head and shoulders, upper body, etc. Common target detection methods include YOLO, SSD / FastRcnn, etc.

[0064] Secondly, based on the detection of human targets, behavior recognition is achieved by adopting a key point + posture classification method. Commonly used methods include: OpenPose, Convolutional Pose Machines, Hourglass, etc.

[0065] Step S4, spatial calibration of panoramic camera and detail camera: realize the coordinate mapping of the two cameras, and calibrate each preset point image with the panoramic camera image. Commonly used multi-camera calibration methods include multi-point calibration and Kalibr-based multi-camera calibration method.

[0066] Step S5, detail camera person identification filtering: The position of the invigilator output in step S2, such as the head and shoulder frame coordinates, is converted and mapped to the detail camera image in step S4 and marked as frame A. Frame A is compared with all target frames detected in step S3, and the target with the highest overlap rate is the invigilator. The corresponding behavior is not reported, and only the analysis results of the examinee's behavior are output.

[0067] The behavior analysis of examinees is achieved through the rotation of detail cameras, which is especially suitable for large-scale classrooms. Through the master-slave linkage method, the invigilators who appear in the detail cameras are excluded, and only the student behavior is counted.

[0068] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0069] In the present embodiment, a device for identifying the behavior of a target object is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0070] Figure 8 is a structural block diagram of an apparatus for identifying target object behavior according to an embodiment of the present invention, such as Figure 8As shown, the device includes: a first acquisition module 802, used to acquire multiple video frames obtained by shooting a target area by a panoramic camera within a target time range, and obtain a first video frame set, wherein the shooting angle of the panoramic camera covers the target area; a second acquisition module 804, used to acquire multiple video frames obtained by shooting N sub-areas included in the target area by a detail camera at N preset points within the target time range, and obtain a second video frame set, wherein N is an integer greater than 1, and the shooting angle of the detail camera at one of the preset points covers one of the sub-areas; an identification module 806, used to obtain a second video frame set through the first video frame set, The second video frame set and the preset pixel position mapping relationship identify the behavior of each target object in the target area, wherein the pixel position mapping relationship records the mapping relationship between the pixel positions in the video frames taken by the panoramic camera and the pixel positions in the video frames taken by the detail camera at each of the preset points, and the target object is an object other than the first object that appears in the target area, and the first object satisfies at least one of the following: the difference between the moving distance of the target object and the moving distance of the first object is greater than a first preset threshold, and the difference between the imaging area of ​​the first object and the imaging area of ​​the target object is greater than a second preset threshold.

[0071] In an exemplary embodiment, the device is also used to input each video frame included in the first video frame set into a first target neural network model in sequence, and identify the first object in the target area through the first target neural network model; input each video frame included in the second video frame set into a second target neural network model in sequence, and identify the objects in the target area and the behavior of each object through the second target neural network model; remove the behavior of the first object from the behaviors of each object identified by the second target neural network model through the preset pixel position mapping relationship to obtain the behavior of the target object.

[0072] In an exemplary embodiment, the device is also used to, when the i-th video frame included in the second video frame set is a video frame shot by the detail camera at the k-th preset point at the j-th time point, input the i-th video frame into the second target neural network model, identify each object in the i-th video frame through the second target neural network model, and obtain a plurality of detection frames of the object, wherein i, j, and k are integers; determine the video frame shot by the panoramic camera at the j-th time point in the first video frame set to obtain the S-th video frame, wherein S is an integer; obtain the first detection frame obtained by the first target neural network model for recognizing the first object in the S-th video frame; and determine the behavior of the target object through the plurality of detection frames, the first detection frame, and the pixel position mapping relationship obtained by the second target neural network model for recognizing the i-th video frame.

[0073] In an exemplary embodiment, the device is also used to determine a second detection frame that matches the first detection frame among the multiple detection frames based on the pixel position mapping relationship; determine the objects in the detection frames other than the second detection frame among the multiple detection frames recognized by the second target neural network model for the i-th video frame as the target objects; and obtain the behavior recognition result of the target object in the i-th video frame by the second target neural network model.

[0074] In an exemplary embodiment, the device is also used to map the first detection frame in the Sth video frame to the i-th video frame according to the pixel position mapping relationship to obtain a first mapped detection frame; and determine the detection frame with the largest overlap rate between the multiple detection frames in the i-th video frame and the first mapped detection frame as the second detection frame.

[0075] In an exemplary embodiment, the device is also used to analyze the moving trajectory of each object in the first video frame set through the trajectory analysis module of the first target neural network model, and determine the object whose difference in moving distance with other objects in the first video frame set is greater than or equal to the first preset threshold as the first target object; identify the identity of the first target object through the identity recognition module of the first target neural network model to obtain the identity tag of the first target moving object; when the identity tag of the first target moving object is the target identity tag, determine the first target moving object as the first object; or analyze the imaging area of ​​each object in the first video frame set through the imaging area analysis module of the first target neural network model, and determine the object whose difference in imaging area with other objects in the first video frame set is greater than or equal to the second preset threshold as the second target object; identify the identity of the second target moving object through the identity recognition module of the first target neural network model to obtain the identity tag of the second target moving object; when the identity tag of the second target moving object is the target identity tag, determine the second target moving object as the first object; after the first object in the target area is identified by the first target neural network model, the position information of the first object is displayed in real time in the video frame obtained by shooting the target area by the panoramic camera.

[0076] In an exemplary embodiment, the apparatus is also used to display the behavior recognition results of each target object on a target display device; when the behavior recognition results of each target object indicate the presence of abnormal behavior, the target position of the target object with the abnormal behavior in the target area is determined based on the detection box obtained by the second target neural network model for recognizing the target object, and a prompt message is sent to the terminal device at the target position.

[0077] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0078] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0079] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0080] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0081] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0082] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the steps of the method described in each embodiment of the present application when executed by a processor.

[0083] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0084] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for identifying target object behavior, characterized in that: include: Acquire multiple video frames obtained by shooting a target area with a panoramic camera within a target time range to obtain a first video frame set, wherein the shooting angle of the panoramic camera covers the target area; Acquire a plurality of video frames obtained by respectively photographing N sub-areas included in the target area at N preset points by a detail camera within the target time range to obtain a second video frame set, wherein N is an integer greater than 1, and the shooting angle of view of the detail camera at one of the preset points covers one of the sub-areas; The behavior of each target object in the target area is identified by the first video frame set, the second video frame set, and a preset pixel position mapping relationship, wherein the pixel position mapping relationship records the mapping relationship between the pixel positions in the video frames taken by the panoramic camera and the pixel positions in the video frames taken by the detail camera at each of the preset points, and the target object is an object other than the first object that appears in the target area, and the first object satisfies at least one of the following: the difference between the moving distance of the target object and the moving distance of the first object is greater than a first preset threshold, and the difference between the imaging area of ​​the first object and the imaging area of ​​the target object is greater than a second preset threshold.

2. The method according to claim 1, characterized in that Identifying behaviors of each target object in the target area by using the first video frame set, the second video frame set, and a preset pixel position mapping relationship includes: Inputting each video frame included in the first video frame set into a first target neural network model in sequence, and identifying the first object in the target area through the first target neural network model; Inputting each video frame included in the second video frame set into a second target neural network model in sequence, and identifying objects in the target area and behaviors of each of the objects through the second target neural network model; The behavior of the target object is obtained by removing the behavior of the first object from the behaviors of each of the objects identified by the second target neural network model through the preset pixel position mapping relationship.

3. The method according to claim 2, characterized in that The method of removing the behavior of the first object from the behaviors of the objects identified by the second target neural network model through the preset pixel position mapping relationship to obtain the behavior of the target object includes: In the case where the i-th video frame included in the second video frame set is a video frame shot by the detail camera at the k-th preset point at the j-th time point, the i-th video frame is input into the second target neural network model, and each object in the i-th video frame is identified by the second target neural network model to obtain a plurality of detection frames of the objects, wherein i, j, and k are integers; Determine the video frame captured by the panoramic camera at the j-th time point in the first video frame set to obtain the S-th video frame, where S is an integer; Obtaining a first detection frame obtained by the first target neural network model recognizing the first object in the Sth video frame; The behavior of the target object is determined by using the multiple detection frames, the first detection frame, and the pixel position mapping relationship obtained by recognizing the i-th video frame through the second target neural network model.

4. The method according to claim 3, characterized in that Determining the behavior of the target object by using the second target neural network model to identify the i-th video frame, the first detection frame, and the preset pixel position mapping relationship, including: Determine a second detection frame matching the first detection frame from the plurality of detection frames according to the pixel position mapping relationship; Determine an object in a detection frame other than the second detection frame among the multiple detection frames obtained by the second target neural network model recognizing the i-th video frame as the target object; Obtain the behavior recognition result of the second target neural network model on the target object in the i-th video frame.

5. The method according to claim 4, characterized in that Determining a second detection frame matching the first detection frame from among the plurality of detection frames according to the pixel position mapping relationship includes: According to the pixel position mapping relationship, mapping the first detection frame in the S-th video frame to the i-th video frame to obtain a first mapped detection frame; The detection frame with the largest overlap rate with the first mapped detection frame among the multiple detection frames in the i-th video frame is determined as the second detection frame.

6. The method according to claim 2, characterized in that Inputting each video frame of the first video frame set into a first target neural network model in sequence, and identifying a first object in the target area through the first target neural network model, comprising: Analyzing the movement trajectory of each object in the first video frame set by the trajectory analysis module of the first target neural network model, and determining an object in the first video frame set whose difference in movement distance with other objects is greater than or equal to the first preset threshold as a first target object; The identity of the first target object is identified by the identity recognition module of the first target neural network model to obtain an identity tag of the first target moving object; when the identity tag of the first target moving object is a target identity tag, the first target moving object is determined to be the first object; or Analyzing the imaging area of ​​each object in the first set of video frames by the imaging area analysis module of the first target neural network model, and determining as a second target object an object whose difference between the imaging area of ​​the first set of video frames and other objects is greater than or equal to the second preset threshold; Identify the identity of the second target mobile object through the identity recognition module of the first target neural network model to obtain an identity tag of the second target mobile object; if the identity tag of the second target mobile object is a target identity tag, determine the second target mobile object as the first object; After identifying the first object in the target area by the first target neural network model, the method further includes: The position information of the first object is displayed in real time in a video frame obtained by photographing the target area with the panoramic camera.

7. The method according to claim 4, characterized in that After obtaining the behavior recognition result of the second target neural network model on the target object in the i-th video frame, the method further includes at least one of the following: Displaying the behavior recognition results of each target object on a target display device; When the behavior recognition results of each of the target objects indicate the presence of abnormal behavior, the target position of the target object in the target area where the abnormal behavior occurs is determined based on the detection frame obtained by recognizing the target object using the second target neural network model, and a prompt message is sent to the terminal device at the target position.

8. A device for identifying target object behavior, characterized in that: include: A first acquisition module is used to acquire a plurality of video frames obtained by shooting a target area by a panoramic camera within a target time range to obtain a first video frame set, wherein the shooting angle of the panoramic camera covers the target area; A second acquisition module is used to acquire a plurality of video frames obtained by respectively photographing N sub-areas included in the target area at N preset points by a detail camera within the target time range to obtain a second video frame set, wherein N is an integer greater than 1, and the shooting angle of view of the detail camera at one of the preset points covers one of the sub-areas; An identification module is used to identify the behavior of each target object in the target area through the first video frame set, the second video frame set, and a preset pixel position mapping relationship, wherein the pixel position mapping relationship records the mapping relationship between the pixel positions in the video frames taken by the panoramic camera and the pixel positions in the video frames taken by the detail camera at each of the preset points, and the target object is an object other than the first object that appears in the target area, and the first object satisfies at least one of the following: the difference between the moving distance of the target object and the moving distance of the first object is greater than a first preset threshold, and the difference between the imaging area of ​​the first object and the imaging area of ​​the target object is greater than a second preset threshold.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • A method and a system for analyzing human behavior of an educational examination monitoring video

    CN109271886A

  • Gun-dome camera linkage target tracking method and device, electronic device and storage medium

    CN114120165A

  • Examinee abnormal behavior detection method based on deep learning

    CN114333070A

  • Target identification method, target identification system, target identification device, equipment and storage medium

    CN118042074A

  • Real-time multi-video stream fusion system in online invigilation

    CN118509552A