A behavior detection method, device and computer readable storage medium
Through the combination of deep learning models and multiple camera devices, efficient violation detection is achieved, solving the problem of time-consuming detection of violations in the prior art, and improving detection efficiency and accuracy.
Patent Information
- Application Number
- CN202010833023.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-18
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-08-18
AI Technical Summary
The existing methods of violation detection take a long time and have low detection efficiency, so it is impossible to efficiently determine whether food processors have violated their violations.
Using a behavior detection method based on a deep learning model, by obtaining the target human image collected by the visible light camera device, inputting a pre-constructed behavior detection model to generate a behavior probability matrix. If the matrix element value is greater than the preset probability threshold, it is determined that there is a violation. Combined with thermal imaging and visible light camera devices to detect violations.
It significantly shortens the behavior detection time and improves detection efficiency, and does not need to compare with a large number of human images, so that violations and identity information can be quickly judged.
Smart Images

Figure CN112084882B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to a behavior detection method, device and computer-readable storage medium. Background Art
[0002] As people's living standards continue to improve, people pay more and more attention to food quality and safety issues. In order to start from the source of food, cameras can usually be installed in food processing rooms to monitor whether food processors have any violations that are not conducive to food safety, such as not wearing masks, gloves or hats. The traditional method of detecting violations is to compare the human body images captured by the camera with a large number of pre-stored human body images of human bodies with violations to determine whether the food processor has any violations and the type of violations. However, the above detection method is time-consuming and the detection efficiency of violations is low. Summary of the invention
[0003] The embodiments of the present application provide a behavior detection method, device, and computer-readable storage medium, which can solve the problems of long time consumption and low efficiency of detecting illegal behaviors in existing illegal behavior detection methods.
[0004] In a first aspect, an embodiment of the present application provides a behavior detection method, comprising:
[0005] Acquire a target human body image captured by a visible light camera device;
[0006] The target human body image is input into a pre-constructed behavior detection model for processing to obtain a behavior probability matrix corresponding to the target human body image; wherein the value of each element in the behavior probability matrix is used to represent the probability that the target human body in the target human body image has a preset violation behavior corresponding to the element, and the behavior detection model is obtained by training a pre-constructed first deep learning model based on a first preset sample set; each sample data in the first preset sample set includes a sample human body image and a behavior probability matrix corresponding to the sample human body image;
[0007] If the value of the first element in the behavior probability matrix is greater than a preset probability threshold corresponding to the first element, it is determined that the target human body has a preset violation behavior corresponding to the first element.
[0008] Furthermore, the step of acquiring the target human body image captured by the visible light camera device includes:
[0009] Acquire multiple frames of human body images captured by the visible light camera device;
[0010] Inputting the multiple frames of human body images into a preset face detection model to obtain face score values corresponding to each of the multiple frames of human body images; wherein the face score values are used to indicate the integrity of the faces in the human body images;
[0011] The human body image with the largest face score value is determined as the target human body image.
[0012] Furthermore, before acquiring the target human body image captured by the visible light camera device, the behavior detection method further includes:
[0013] Acquire multiple frames of video images captured by the visible light camera device;
[0014] Inputting the multiple frames of video images into a preset target detection model for target recognition, and obtaining target recognition results corresponding to each of the multiple frames of video images;
[0015] The video image including a human body in the target recognition result is determined as the human body image.
[0016] Furthermore, after determining that the target human body has a preset illegal behavior corresponding to the first element, the behavior detection method further includes:
[0017] Acquire feature information of a human face contained in the target human image;
[0018] The identity information of the target human body is determined according to the feature information of the human face contained in the target human body image and the correspondence between the feature information of each preset human face and the identity information stored in advance.
[0019] Furthermore, the behavior detection method also includes:
[0020] Acquire a first thermal image captured by a thermal imaging camera;
[0021] If there is a first area in the first thermal image whose temperature value is greater than a first preset temperature threshold, determining that a first human body detected by the first thermal image has a first preset violation, and determining coordinate information of the first area;
[0022] Acquire a first human body image captured by a target visible light camera device associated with the thermal imaging camera device; wherein the target visible light camera device and the thermal imaging camera device are arranged at the same position, and the acquisition time of the first human body image is the same as the acquisition time of the first thermal image;
[0023] Determining, according to the coordinate information of the first area, coordinate information of a second area in the first human body image corresponding to the first area;
[0024] Performing face detection on the first human image, and determining coordinate information of each candidate human face contained in the first human image;
[0025] Determining a distance value between each of the candidate human faces and the second region according to the coordinate information of each of the candidate human faces and the coordinate information of the second region;
[0026] The identity information of the first human body is determined according to the feature information of the candidate human face having the smallest distance value to the second area.
[0027] Furthermore, the behavior detection method also includes:
[0028] Acquire a second thermal image captured by the thermal imaging camera device within a preset time period;
[0029] If there is a third area in the second thermal image whose temperature is greater than the second preset temperature threshold and less than the first preset temperature threshold, it is determined that there is a preset target object in the target objects detected by the second thermal image.
[0030] Furthermore, after determining that the target human body has a preset illegal behavior corresponding to the first element, the behavior detection method further includes:
[0031] Output an alarm control instruction to a preset alarm device, wherein the alarm control instruction is used to instruct the alarm device to sound an alarm.
[0032] In a second aspect, an embodiment of the present application provides a behavior detection device, including:
[0033] A first acquisition unit, used to acquire a target human body image captured by a visible light camera device;
[0034] A behavior detection unit is used to input the target human body image into a pre-constructed behavior detection model for processing to obtain a behavior probability matrix corresponding to the target human body image; wherein the value of each element in the behavior probability matrix is used to represent the probability that the target human body in the target human body image has a preset violation behavior corresponding to the element, and the behavior detection model is obtained by training a pre-constructed first deep learning model based on a first preset sample set; each sample data in the first preset sample set includes a sample human body image and a behavior probability matrix corresponding to the sample human body image;
[0035] The behavior determination unit is used to determine that the target human body has a preset violation behavior corresponding to the first element if the value of the first element in the behavior probability matrix is greater than a preset probability threshold corresponding to the first element.
[0036] In a third aspect, an embodiment of the present application provides a behavior detection device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the behavior detection method as described in any one of the first aspects above when executing the computer program.
[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the behavior detection method as described in any one of the above-mentioned first aspects are implemented.
[0038] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a behavior detection device, the behavior detection device executes the steps of the behavior detection method described in any one of the first aspects above.
[0039] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0040] The present application provides a behavior detection method, which obtains a target human body image collected by a visible light camera device; inputs the target human body image into a pre-constructed behavior detection model for processing to obtain a behavior probability matrix corresponding to the target human body image; wherein the value of each element in the behavior probability matrix is used to represent the probability that the target human body in the target human body image has a preset violation behavior corresponding to the element, and the behavior detection model is obtained by training a pre-constructed first deep learning model based on a first preset sample set; each sample data in the first preset sample set includes a sample human body image and a behavior probability matrix corresponding to the sample human body image; if the value of the first element in the behavior probability matrix is greater than the preset probability threshold corresponding to the first element, it is determined that the target human body has a preset violation behavior corresponding to the first element. The above-mentioned behavior detection method only needs to input the target human body image collected by the visible light camera device into the pre-constructed behavior detection model to obtain the behavior probability matrix corresponding to the target human body image, and then can determine whether the target human body has a violation behavior or the type of violation behavior according to the value of each element in the behavior probability matrix and the preset probability threshold corresponding to each element. Compared with the prior art, the present application does not need to compare the target human body image with a large number of human body images with human violations, thereby shortening the time of behavior detection and improving the efficiency of behavior detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0042] Figure 1 is a system schematic diagram of a behavior detection system provided in an embodiment of the present application;
[0043] Figure 2 It is a structural diagram of a behavior detection model provided in an embodiment of the present application;
[0044] Figure 3 It is a flow chart of an implementation of a behavior detection method provided in an embodiment of the present application;
[0045] Figure 4 This is a specific implementation flow chart of S101 in a behavior detection method provided in an embodiment of the present application;
[0046] Figure 5 is a flow chart of an implementation of a behavior detection method provided by another embodiment of the present application;
[0047] Figure 6 is a flow chart of an implementation of a behavior detection method provided in yet another embodiment of the present application;
[0048] Figure 7 is a flow chart of an implementation of a behavior detection method provided by another embodiment of the present application;
[0049] Figure 8 is a schematic diagram of the coordinate range of the first area and the second area provided in an embodiment of the present application;
[0050] Fig. 9 is a flow chart of an implementation of a behavior detection method provided by another embodiment of the present application;
[0051] Fig.10 is a schematic diagram of the structure of a behavior detection device provided in an embodiment of the present application;
[0052] Fig.11 It is a structural diagram of a behavior detection device provided in another embodiment of the present application. DETAILED DESCRIPTION
[0053] The following embodiments of this application are only used in legal situations.
[0054] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0055] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0056] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0057] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0058] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0059] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0060] Figure 1 is a schematic diagram of the structure of a behavior detection system applicable to a behavior detection method provided in an embodiment of the present application, such as Figure 1As shown, the behavior detection system may include a behavior detection device 10 , and a thermal imaging camera 11 and at least one visible light camera 12 that are communicatively connected to the behavior detection device 10 .
[0061] The visible light camera 12 is used to capture objects within its camera range to obtain corresponding red, green, and blue (RGB) images.
[0062] The thermal imaging camera 11 detects infrared radiation from the object under test within its shooting range, and converts the invisible infrared energy emitted by the object under test into a visible thermal image by means of signal processing, photoelectric conversion, etc. Different colors in the thermal image represent different temperatures of the object under test.
[0063] The behavior detection device 10 can obtain the RGB image collected by the visible light camera device 12 and / or the thermal image collected by the thermal imaging camera device 11, and detect the behavior of the person being photographed based on the RGB image and / or the thermal image. Therefore, the visible light camera device and the thermal imaging camera device can be installed in any place where behavior detection is required, such as various shopping malls, supermarkets or food processing rooms.
[0064] For example, taking the application of the behavior detection system in the field of food safety as an example, in order to ensure the safety of food processing, a thermal imaging camera 11 and multiple visible light cameras 12 can be installed in the food processing room. Among them, different visible light cameras 12 can be installed in different positions in the food processing room, and the thermal imaging camera 11 can be installed in the same position as one of the visible light cameras 12. The behavior detection device 10 can associate the visible light camera 12 and the thermal imaging camera 11 installed in the same position.
[0065] In practical applications, as an example but not limitation, the behavior detection device 10 may be a desktop computer, a notebook, a handheld computer, etc. The thermal imaging camera device 11 may be an infrared thermal imaging thermometer. The visible light camera device 12 may be a monitoring camera device.
[0066] It should be noted that in the embodiment of the present application, a plurality of detection models are pre-stored in the behavior detection device 10, including but not limited to: a behavior detection model, a target detection model, and a face detection model.
[0067] The behavior detection model is used to detect the probability of a preset violation of the target human body in the human body image. Wherein, the human body image refers to an image containing a human body (at least including the upper body of the human body). The preset violation can be set according to actual needs, and there is no restriction here. For example, when the behavior detection model is applied to the field of food safety, it is used to detect whether the food processors in the food processing room have preset violation operations that are not conducive to food safety. The preset violation may include but is not limited to not wearing a mask, not wearing a hat, not wearing gloves, not wearing work clothes and smoking. The behavior detection model can be obtained by training a pre-constructed first deep learning model based on a first preset sample set. Wherein, each sample data in the first preset sample set includes a sample human body image and a behavior probability matrix corresponding to the sample human body image. The value of each element in the behavior probability matrix is used to represent the probability that the target human body in the sample human body image has a preset violation of the corresponding element. When training the pre-constructed first deep learning model, the sample human body image in each sample is used as the input of the first deep learning model, and the behavior probability matrix corresponding to the sample human body image in each sample is used as the output of the first deep learning model. Through training, the first deep learning model can learn the correspondence between all possible human body images and behavior probability matrices, and the trained first deep learning model is used as the behavior detection model.
[0068] Exemplarily, assuming that the behavior probability matrix is [A, B, C], element A, element B and element C correspond to preset violation 1, preset violation 2 and preset violation 3 respectively, then the value of element A is used to represent the probability that the target human body in the target human body image has preset violation 1, the value of element B is used to represent the probability that the target human body in the target human body image has preset violation 2, and the value of element C is used to represent the probability that the target human body in the target human body image has preset violation 3.
[0069] In the embodiment of the present application, the behavior detection model can be composed of a plurality of sub-behavior detection models for detecting different illegal behaviors. Among them, the preset illegal behaviors can be set according to actual needs and are not limited here.
[0070] Based on this, Figure 2 As shown, the behavior detection model 20 may include a first sub-behavior detection model 21 for detecting not wearing a mask, a second sub-behavior detection model 22 for detecting not wearing work clothes, and a third sub-behavior detection model 23 for detecting not wearing a work hat.
[0071] The target detection model is used to detect the target object in the image and identify the type of the target object. The target detection model can be an existing target detection model based on a convolutional neural network.
[0072] The face detection model is used to detect the integrity of the face contained in the human body image. The face detection model can be obtained by training the pre-constructed second deep learning model based on the second preset sample set. Among them, each sample data in the second preset sample set includes a sample human body image and a face score value corresponding to the sample human body image. The face score value is used to represent the completeness and clarity of the facial features in the face of the target human body in the sample human body image. The higher the score value, the more complete the facial features are and the higher the clarity of the facial features. When training the pre-constructed second deep learning model, the sample human body image in each sample is used as the input of the second deep learning model, and the face score value corresponding to the sample human body image in each sample is used as the output of the second deep learning model. Through training, the second deep learning model can learn the correspondence between all possible human body images and face score values, and the trained second deep learning model is used as the face detection model.
[0073] The behavior detection method provided in the embodiment of the present application is exemplarily described below by taking the installation of the visible light camera device 12 and the thermal imaging camera device 11 in a food processing room as an example.
[0074] See also Figure 3 , Figure 3 is a flow chart of an implementation of a behavior detection method provided in an embodiment of the present application. The execution subject of the behavior detection method provided in an embodiment of the present application is a behavior detection device. Figure 3 As shown, the behavior detection method may include S101 to S103, which are described in detail as follows:
[0075] In S101, a target human body image captured by a visible light camera device is obtained.
[0076] In the embodiment of the present application, the visible light camera device can record the scene within its shooting range to obtain a corresponding video. Alternatively, the visible light camera device can take pictures of the scene within its shooting range based on a preset shooting time interval to obtain multiple frames of video images.
[0077] The behavior detection device can obtain the video captured by the visible light camera device, and perform frame processing on the video to obtain multiple frames of video images. Alternatively, the visible light camera device can directly obtain the multiple frames of video images captured by the visible light camera device.
[0078] After the visible light camera device acquires the multiple frames of video images captured by the visible light camera device, the target human body image is acquired from the multiple frames of video images.
[0079] Specifically, the behavior detection device can be Figure 4 S401 to S403 shown in the figure obtain a target human body image, which is described in detail as follows:
[0080] In S401, multiple frames of human body images captured by the visible light camera device are acquired.
[0081] In this embodiment, since there may be human bodies or non-human bodies within the shooting range of the visible light camera device, the multiple frames of video images captured by the visible light camera device may include human bodies or non-human bodies. The non-human bodies may be living beings other than humans or inanimate objects.
[0082] Based on this, the behavior detection device can be Figure 5 S01 to S03 shown determine the human body image collected by the visible light camera device, as detailed as follows:
[0083] In S01, multiple frames of video images captured by the visible light camera device are acquired.
[0084] In S02, the multiple frames of video images are input into a preset target detection model for target recognition, and target recognition results corresponding to each of the multiple frames of video images are obtained.
[0085] The target recognition result is used to describe the type of target object contained in the video image.
[0086] The type of the target object may include but is not limited to a human body or a non-human body. For example, a non-human body may include but is not limited to a cat, a dog, a mouse, or a tableware.
[0087] If the behavior detection device detects that the target recognition result corresponding to a frame of video image includes a human body, S03 is executed.
[0088] If the behavior detection device detects that the target recognition result corresponding to a certain frame of video image includes a life form other than a human, the behavior detection device can output a first alarm control instruction to a preset alarm device connected to the behavior detection device to control the preset alarm device to alarm in a first alarm mode. The first alarm mode is used to prompt that there is a life form other than a human in the food processing room. As an example and not a limitation, the alarm device can be an alarm or an indicator light.
[0089] In S03, the video image including a human body in the target recognition result is determined as the human body image.
[0090] When the target recognition result corresponding to a frame of video image includes a human body, it indicates that the video image includes a human body. Therefore, the behavior detection device can determine the video image including a human body in the target recognition result as a human body image.
[0091] In S402, the multiple frames of human body images are input into a preset face detection model to obtain face score values corresponding to each of the multiple frames of human body images; wherein the face score values are used to indicate the integrity of the faces in the human body images.
[0092] In S403, the human body image with the largest face score is determined as the target human body image.
[0093] In this embodiment, the behavior detection device may determine the human body image with the largest face score as the target human body image.
[0094] In another embodiment of the present application, the behavior detection device may also determine a human image whose face score value is greater than a preset face score value threshold as a target human image. The human image whose face score value is greater than the preset face score value threshold may be one frame or multiple frames.
[0095] In S102, the target human body image is input into a pre-constructed behavior detection model for processing to obtain a behavior probability matrix corresponding to the target human body image; wherein the value of each element in the behavior probability matrix is used to represent the probability that the target human body in the target human body image has a preset violation behavior corresponding to the element, and the behavior detection model is obtained by training a pre-constructed first deep learning model based on a first preset sample set; each sample data in the first preset sample set includes a sample human body image and a behavior probability matrix corresponding to the sample human body image.
[0096] After obtaining the behavior probability matrix corresponding to the target human image, the behavior detection device can compare the value of each element in the behavior probability matrix with the preset probability threshold corresponding to the element. If the behavior detection device detects that the value of at least one element in the behavior probability matrix is greater than the preset probability threshold corresponding to the element, S103 is executed; if the behavior detection device detects that the value of each element in the behavior probability matrix is less than or equal to the preset probability threshold corresponding to the element, it is determined that the target human body in the target human image does not have any illegal behavior.
[0097] It should be noted that the preset probability threshold corresponding to each element may be the same or different, and the preset probability threshold may be set according to actual needs and is not limited here.
[0098] The behavior detection device inputs the target human image into a pre-built behavior detection model for processing, which may specifically include: inputting the target human image into a plurality of sub-behavior detection models for detecting different violations, each sub-behavior detection model outputs a probability value, and then obtains a behavior probability matrix composed of each probability value. Each probability value is used to indicate the probability of a certain preset violation in the target human image.
[0099] In S103, if the value of the first element in the behavior probability matrix is greater than a preset probability threshold corresponding to the first element, it is determined that the target human body has a preset violation behavior corresponding to the first element.
[0100] Exemplarily, assuming that the value of the behavior probability matrix [A, B, C] is [0.9, 0.7, 0.95], and the preset probability threshold corresponding to each element is 0.9, since the probability value of element C is greater than 0.9, the behavior detection device can determine that the target human body in the target human body image has the preset violation behavior 3 corresponding to element C.
[0101] In one embodiment of the present application, after detecting that the target human body has a preset violation, the behavior detection device can output a second alarm control instruction to a preset alarm device connected to the behavior detection device to control the preset alarm device to alarm in a second alarm mode. The second alarm mode is used to prompt that the target human body has a preset violation.
[0102] As can be seen from the above, a behavior detection method provided by an embodiment of the present application obtains a target human body image collected by a visible light camera device; the target human body image is input into a pre-constructed behavior detection model for processing to obtain a behavior probability matrix corresponding to the target human body image; wherein the value of each element in the behavior probability matrix is used to represent the probability that the target human body in the target human body image has a preset violation behavior corresponding to the element, and the behavior detection model is obtained by training a pre-constructed first deep learning model based on a first preset sample set; each sample data in the first preset sample set includes a sample human body image and a behavior probability matrix corresponding to the sample human body image; if the value of the first element in the behavior probability matrix is greater than the preset probability threshold corresponding to the first element, it is determined that the target human body has a preset violation behavior corresponding to the first element. The above behavior detection method only needs to input the target human body image collected by the visible light camera device into the pre-constructed behavior detection model to obtain the behavior probability matrix corresponding to the target human body image, and then it can be determined whether the target human body has a violation behavior or the type of violation behavior according to the value of each element in the behavior probability matrix and the preset probability threshold corresponding to each element. Compared with the prior art, the present application does not need to compare the target human body image with a large number of human body images with human violations, thereby shortening the behavior detection time and improving the efficiency of behavior detection.
[0103] In another embodiment of the present application, in order to determine the identity information of the target person in the target person image with illegal behavior, the behavior detection device can pre-acquire the facial image and identity information of each food processor, and extract facial feature information from the facial image of each food processor as the feature information of the face of the food processor. The behavior detection device can establish a correspondence between the feature information of the face of each food processor and its identity information, and obtain the correspondence between the feature information and identity information of each preset face. Among them, each preset face corresponds to the face of each food processor. The identity information may include but is not limited to: name and position, etc. The behavior detection device can store the correspondence between the feature information and identity information of each preset face.
[0104] Based on this, Figure 6 FIG. 1 is a flowchart of a behavior detection method provided in another embodiment of the present application. Figure 3 Corresponding to the embodiment, the behavior detection method provided in this embodiment may further include the following steps after S103: Figure 6 S601 to S602 shown are described in detail as follows:
[0105] In S601, feature information of a human face contained in the target human image is obtained.
[0106] In this embodiment, after determining that the target human body has the preset violation corresponding to the first element, feature information of the face of the target human body is extracted from the target human body image. As an example but not limitation, the feature information of the face can be feature information of a preset key area in the face, and as an example but not limitation, the preset key area can be a nose, eyes, mouth and other areas. It can be understood that in this embodiment, the preset key area can be one or more.
[0107] In S602, the identity information of the target person is determined according to the feature information of the face contained in the target person image and the pre-stored correspondence between the feature information of each preset face and the identity information.
[0108] In this embodiment, the behavior detection device can compare the characteristic information of the face contained in the target human image with the characteristic information of each pre-stored preset face one by one. If it is detected that the similarity between the characteristic information of the face contained in the target human image and the characteristic information of the first pre-stored preset face is greater than or equal to the preset similarity threshold, the identity information corresponding to the characteristic information of the first preset face is determined as the identity information corresponding to the characteristic information of the face contained in the target human image, and the characteristic information of the face contained in the target human image is the identity information of the target human body. Among them, the preset similarity threshold can be set according to actual needs and is not limited here.
[0109] It should be noted that, in the present embodiment, the similarity between the feature information of the human face contained in the target human image and the feature information of a certain preset human face stored in advance is greater than or equal to a preset similarity threshold, specifically referring to the similarity between the feature information of each preset key area of the human face contained in the target human image and the feature information of the corresponding preset key area contained in a certain preset human face stored in advance is greater than or equal to the preset similarity threshold.
[0110] In another embodiment of the present application, the behavior detection device determines that the target person is a non-food processor after detecting that the similarity between the facial feature information contained in the target person image and the pre-stored facial feature information is less than a preset similarity threshold.
[0111] Based on this, in another embodiment of the present application, if the behavior detection device detects that the target human body with illegal behavior is not a food processor, it can output a third alarm control instruction to a preset alarm device connected to the behavior detection device to control the preset alarm device to alarm in a third alarm mode. The third alarm mode is used to prompt that the target human body with illegal behavior in the food processing room is not a food processor.
[0112] From the above, it can be seen that a behavior detection method provided by this embodiment obtains the feature information of the human face contained in the target human image; determines the identity information of the target human body according to the feature information of the human face contained in the target human body image and the correspondence between the feature information and identity information of each pre-stored preset human face, so that the behavior detection method can not only determine whether there are any violations stored in the food processing room, but also determine the identity information of the target human body that has committed the violations.
[0113] See also Figure 7 , Figure 7 FIG. 1 is a flowchart of a behavior detection method provided by another embodiment of the present application. Figure 3 Corresponding to the embodiment, a behavior detection method in this embodiment may further include S701 to S707, which are described in detail as follows:
[0114] In S701, a first thermal image captured by a thermal imaging camera is acquired.
[0115] In this embodiment, after the thermal imaging camera device is turned on, the corresponding thermal image of the food processing room can be obtained in real time. The thermal image is used to describe the heat or temperature of objects within the shooting range of the thermal imaging camera device.
[0116] It should be noted that, since the temperature of a cigarette butt is generally higher than that of a human body, the behavior detection device can determine whether there is smoking behavior in the food processing room based on different temperature values in the thermal image.
[0117] Based on this, the behavior detection device can obtain the first thermal image collected by the thermal imaging camera device. The first thermal image can be any one or more thermal images collected by the thermal imaging camera device.
[0118] After acquiring the first thermal image, the behavior detection device may compare the temperature values of all pixels in the first thermal image with the first preset temperature threshold. If the behavior detection device detects that there is a first area in the first thermal image whose temperature value is greater than the first preset temperature threshold, S702 to S707 are executed; if the behavior detection device detects that there is no first area in the first thermal image whose temperature value is greater than the first preset temperature threshold, it is determined that the first human body detected by the first thermal image does not have the first preset violation behavior.
[0119] The first preset temperature threshold can be set according to actual needs, and is not limited here. For example, the first preset temperature threshold can be the temperature value after the tobacco is ignited. The first preset violation behavior can be smoking behavior.
[0120] In S702, if there is a first area in the first thermal image whose temperature value is greater than a first preset temperature threshold, it is determined that a first human body detected by the first thermal image has a first preset violation, and coordinate information of the first area is determined.
[0121] It should be noted that the coordinate information of the first area refers to the coordinate range of the first area. As an example but not a limitation, the first area may be a rectangular area, and the coordinate range of the first area may be represented by the coordinates of the two endpoints of any diagonal line of the rectangular area.
[0122] In this embodiment, the coordinate information of the first area is based on the first coordinate system corresponding to the first thermal image. Figure 8 As shown in (a) in FIG. 1 , the behavior detection device can establish a first coordinate system with the endpoint A of the first thermal image 30 as the origin, the straight line where the left vertical side 31 of the first thermal image 30 is located as the X-axis, and the straight line where the lower horizontal side 32 of the first thermal image 30 is located as the Y-axis. Based on this, the coordinate information of the first area refers to the coordinate range of the first area in the first coordinate system.
[0123] For example, if the coordinates of the first endpoint C1 of the first region 34 are (x1, y1), and the coordinates of the second endpoint C2 on the same diagonal line as the first endpoint C1 are (x2, y2), then the coordinate range of the first region 34 can be expressed as: [(x1, y1), (x2, y2)] , That is, the coordinate information of the first area 34 is: [(x1, y1), (x2, y2)].
[0124] In S703, a first human body image captured by a target visible light camera device associated with the thermal imaging camera device is obtained; wherein the target visible light camera device and the thermal imaging camera device are set at the same position, and the acquisition time of the first human body image is the same as the acquisition time of the first thermal image.
[0125] In this embodiment, since the thermal imaging camera device can only obtain thermal images and cannot obtain corresponding RGB images, in order to determine the identity information of the human body that has the first preset violation behavior, the behavior detection device can obtain the first human body image captured by the target visible light camera device associated with the thermal imaging camera device. The acquisition time of the first human body image should be the same as the acquisition time of the first thermal image.
[0126] In S704, coordinate information of a second area corresponding to the first area in the first human body image is determined according to the coordinate information of the first area.
[0127] In one implementation of the present embodiment, when the thermal imaging camera device and the target visible light camera device have the same field of view angle, that is, the shooting range of the thermal imaging camera device and the target visible light camera device is the same, the scene in the first thermal image taken by the thermal imaging camera device is completely consistent with the scene in the first human body image taken by the target visible light camera device. Therefore, the position of the first area in the first thermal image is exactly the same as the position of the second area corresponding to the first area in the first human body image.
[0128] The position of the second region in the first human body image can be represented by coordinate information of the second region relative to a second coordinate system, wherein the second coordinate system is established in the same manner as the first coordinate system.
[0129] like Figure 8 As shown in (b) in FIG. 7 , the behavior detection device can establish a second coordinate system with the endpoint B of the first human image 40 as the origin, the straight line where the left vertical side 41 of the first human image 40 is located as the X-axis, and the straight line where the lower horizontal side 42 of the first human image 40 is located as the Y-axis. Since the coordinate information of the second area relative to the second coordinate system is the same as the coordinate information of the first area relative to the first coordinate system, therefore, in combination with S702, if the coordinate information of the first area is: [(x1, y1), (x2, y2)], the coordinate information of the second area 44 is also: [(x1, y1), (x2, y2)].
[0130] In another implementation of this embodiment, when the field of view of the thermal imaging camera device and the target visible light camera device are different, that is, the shooting range of the thermal imaging camera device and the target visible light camera device are not exactly the same, the position of the same object in the food processing room in the images shot by the thermal imaging camera device and the target visible light camera device is different. Therefore, the behavior detection device can use the two reference objects that can be photographed by the thermal imaging camera device and the target visible light camera device as references to respectively determine the coordinate information of the reference area formed by the two reference objects relative to the first coordinate system and the second coordinate system. The reference area can be a rectangular area with the line connecting the two reference objects as the diagonal line.
[0131] Based on this, since the coordinate information of the first area relative to the reference area and the coordinate information of the second area relative to the reference area are the same, the behavior detection device can first determine the coordinate information of the reference area in different coordinate systems, and then determine the position of the first area relative to the reference area based on the coordinate information of the first area in the first coordinate system and the coordinate information of the reference area in the first coordinate system, and finally determine the coordinate information of the second area relative to the second coordinate system based on the position of the first area relative to the reference area and the coordinate information of the reference area in the second coordinate system, that is, determine the coordinate information of the second area in the first human body image.
[0132] For example, Figure 8 As shown in (c), assuming that the reference area 33 is in the first thermal image 30 of the thermal imaging camera device, the coordinate information of the first area 34 in the first coordinate system is: [(x1, y1), (x2, y2)], and the coordinate information of the reference area 33 in the first coordinate system is: [(x3, y3), (x4, y4)], then the position of the first area 34 relative to the reference area 33 can be represented by the midpoint O1 of the first area 34 (the coordinate information of the midpoint O1 is: ((x2-x1) / 2, (y1-y2) / 2)) and the midpoint O3 of the reference area 33 (the coordinate information of the midpoint O3 is: ((x4-x3) / 2, (y3-y4) / 2)), that is, the first area 34 is located at the position of the reference area 33 translated to the left by (x4-x3) / 2-(x2-x1) / 2, and then translated downward by (y3-y4) / 2-(y1-y2) / 2.
[0133] Based on this, Figure 8 As shown in (d), assuming that the first human body image of the target visible light camera device is 40, the coordinate information of the reference area 33 in the second coordinate system is: [(x5, y5), (x6, y6)]. The behavior detection device can determine the coordinate information of the second area 44 relative to the second coordinate system based on the coordinate information of the reference area 33 in the second coordinate system and the position of the first area in the first coordinate system relative to the reference area.
[0134] In S705, face detection is performed on the first human image, and coordinate information of each candidate human face included in the first human image is determined.
[0135] In this embodiment, after determining the coordinate information of the second area in the first human image, the behavior detection device can detect each candidate face in the first human image according to the target detection model, and determine the coordinate information of each candidate face. The coordinate information of the candidate face can be represented by the coordinate information of the rectangular area occupied by the candidate face, and the coordinate information of the rectangular area occupied by the candidate face refers to the coordinate range of the rectangular area occupied by the candidate face relative to the second coordinate system.
[0136] In S706, the distance value between each of the candidate human faces and the second region is determined according to the coordinate information of each of the candidate human faces and the coordinate information of the second region.
[0137] In one embodiment, the distance value between the candidate human face and the second region may be a straight-line distance value between a midpoint of a rectangular region occupied by the candidate human face and a midpoint of the second region.
[0138] For example, Figure 8 As shown, if the midpoint of the second area 44 is O2 and the midpoint of the candidate human face area 45 is O4, the behavior detection device can determine the distance value a1 between the midpoint O2 and the midpoint O4 as the distance value between the candidate human face area 45 and the second area 44.
[0139] After the behavior detection device determines the distance values between each candidate human face and the second region, the candidate human face with the smallest distance value from the second region among the candidate human faces can be determined as the human face of the first human body.
[0140] In S707, the identity information of the first human body is determined based on the feature information of the candidate face having the smallest distance value to the second region.
[0141] In this embodiment, the behavior detection device obtains the feature information of the candidate face with the smallest distance value to the second area, and compares the feature information of the candidate face with the feature information of each pre-stored preset face one by one. If it is detected that the similarity between the feature information of the candidate face and the feature information of the pre-stored second preset face is greater than or equal to the preset similarity threshold, the identity information corresponding to the feature information of the second preset face is determined as the identity information corresponding to the feature information of the candidate face, and the identity information corresponding to the feature information of the candidate face is the identity information of the first person.
[0142] As can be seen from the above, a behavior detection method provided by this embodiment obtains a first thermal image collected by a thermal imaging camera device; if there is a first area in the first thermal image whose temperature value is greater than a first preset temperature threshold, it is determined that the first human body detected by the first thermal image has a first preset violation, and the coordinate information of the first area is determined; the first human body image collected by the target visible light camera device associated with the thermal imaging camera device is obtained; the acquisition time of the first human body image is the same as the acquisition time of the first thermal image; face detection is performed on the first human body image, and the coordinate information of each candidate face contained in the first human body image is determined; according to the coordinate information of the first area, the coordinate information of the second area corresponding to the first area in the first human body image is determined; according to the coordinate information of each candidate face and the coordinate information of the second area, the distance value between each candidate face and the second area is determined; according to the feature information of the candidate face with the smallest distance value to the second area, the identity information of the first human body is determined. The above behavior detection method can determine whether the first human body has the first preset violation according to the thermal imaging camera device, and can determine the identity information of the first human body according to the target visible light camera device associated with the thermal imaging camera device, thereby improving the detection efficiency and accuracy of the first preset violation.
[0143] In another embodiment of the present application, since there is usually no one in the food processing room during the preset time period, and there may be animals such as mice, causing food damage or loss, in order to detect whether there are animals such as mice in the food processing room during the preset time period, the present application embodiment provides another behavior detection method. The preset time period may be 22:00-6:00.
[0144] See also Fig. 9 , Fig. 9 FIG. 1 is a flowchart of a behavior detection method provided by another embodiment of the present application. Figure 3 Corresponding to the embodiment, the behavior detection method provided in this embodiment may also include the following Fig. 9 S901 to S902 shown are described in detail as follows:
[0145] In S901, a second thermal image captured by the thermal imaging camera device within a preset time period is acquired.
[0146] The second thermal image may be any one or more thermal images captured by the thermal imaging camera within a preset time period.
[0147] After the behavior detection device acquires the second thermal image, if it detects that there is a third area in the second thermal image whose temperature is greater than the second preset temperature threshold and less than the first preset temperature threshold, S802 is executed; if the behavior detection device detects that there is no third area in the second thermal image whose temperature is greater than the second preset temperature threshold and less than the first preset temperature threshold, it is determined that there is no preset target object in the food processing room within the preset time period. The second preset temperature threshold is less than the first preset temperature threshold, and the second preset temperature threshold can be set according to actual needs, which is not limited here. For example, the second preset temperature threshold can be the body temperature of a mouse. The preset target object can be a mouse.
[0148] In S902 , if there is a third area in the second thermal image whose temperature is greater than the second preset temperature threshold and less than the first preset temperature threshold, it is determined that a preset target object exists in the target objects detected by the second thermal image.
[0149] In one embodiment of the present application, if the behavior detection device detects that there is a preset target object among the target objects detected by the second thermal image, the behavior detection device can output a fourth alarm control instruction to a preset alarm device connected to the behavior detection device to control the preset alarm device to alarm in a fourth alarm mode. The fourth alarm mode is used to prompt that there is a preset target object in the food processing room within a preset time period.
[0150] As can be seen from the above, the behavior detection method provided by this embodiment obtains the second thermal image collected by the thermal imaging camera device within a preset time period; if there is a third area in the second thermal image whose temperature is greater than the second preset temperature threshold and less than the first preset temperature threshold, it is determined that there is a preset target object in the target object detected by the second thermal image, and an alarm is issued. The above behavior detection method ensures that the food in the food processing room will not be damaged or lost within the preset time period.
[0151] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0152] Corresponding to the real-time clock switching method described in the above embodiment, Fig.10 FIG. 1 is a block diagram of a behavior detection device provided in an embodiment of the present application. For the sake of convenience, only the part related to the embodiment of the present application is shown. Fig.10 The behavior detection device 600 includes: a first acquisition unit 61, a behavior detection unit 62 and a behavior determination unit 63. Among them:
[0153] The first acquisition unit 61 is used to acquire the target human body image captured by the visible light camera device.
[0154] The behavior detection unit 62 is used to input the target human body image into a pre-constructed behavior detection model for processing to obtain a behavior probability matrix corresponding to the target human body image; wherein the value of each element in the behavior probability matrix is used to represent the probability that the target human body in the target human body image has a preset violation behavior corresponding to the element, and the behavior detection model is obtained by training a pre-constructed first deep learning model based on a first preset sample set; each sample data in the first preset sample set includes a sample human body image and a behavior probability matrix corresponding to the sample human body image.
[0155] The behavior determination unit 63 is used to determine that the target human body has a preset violation behavior corresponding to the first element if the value of the first element in the behavior probability matrix is greater than a preset probability threshold corresponding to the first element.
[0156] In one embodiment of the present application, the first acquisition unit specifically includes: a second acquisition unit, a face detection unit and a target image determination unit. Among them:
[0157] The second acquisition unit is used to acquire multiple frames of human body images collected by the visible light camera device.
[0158] The face detection unit is used to input the multiple frames of human body images into a preset face detection model to obtain face score values corresponding to each of the multiple frames of human body images; wherein the face score values are used to indicate the integrity of the faces in the human body images.
[0159] The target image determination unit is used to determine the human body image with the largest face score value as the target human body image.
[0160] Based on this, in one embodiment of the present application, the behavior detection device further includes: a third acquisition unit, a target recognition unit and a human image determination unit. Among them:
[0161] The third acquisition unit is used to acquire multiple frames of video images captured by the visible light camera device.
[0162] The target recognition unit is used to input the multiple frames of video images into a preset target detection model for target recognition, and obtain target recognition results corresponding to each of the multiple frames of video images.
[0163] The human body image determination unit is used to determine the video image including the human body in the target recognition result as the human body image.
[0164] In one embodiment of the present application, the behavior detection device further includes: a fourth acquisition unit and a target human identity information determination unit.
[0165] The fourth acquisition unit is used to acquire feature information of the human face contained in the target human image.
[0166] The target human identity information determination unit is used to determine the identity information of the target human according to the feature information of the human face contained in the target human image and the correspondence between the feature information and identity information of each pre-stored preset human face.
[0167] In one embodiment of the present application, the behavior detection device further includes: a fifth acquisition unit, a violation determination unit, a sixth acquisition unit, a first coordinate information determination unit, a second coordinate information determination unit, a distance value determination unit, and a first human identity information determination unit.
[0168] The fifth acquisition unit is used to acquire the first thermal image captured by the thermal imaging camera device.
[0169] The violation determination unit is used to determine that a first human body detected by the first thermal image has a first preset violation if there is a first area with a temperature value greater than a first preset temperature threshold in the first thermal image, and determine the coordinate information of the first area.
[0170] The sixth acquisition unit is used to acquire a first human body image captured by a target visible light camera device associated with the thermal imaging camera device; wherein the target visible light camera device and the thermal imaging camera device are arranged at the same position, and the acquisition time of the first human body image is the same as the acquisition time of the first thermal image.
[0171] The first coordinate information determining unit is used to determine the coordinate information of a second area corresponding to the first area in the first human body image according to the coordinate information of the first area.
[0172] The second coordinate information determining unit is used to perform face detection on the first human image and determine coordinate information of each candidate human face contained in the first human image.
[0173] The distance value determination unit is used to determine the distance value between each of the candidate human faces and the second area according to the coordinate information of each of the candidate human faces and the coordinate information of the second area.
[0174] The first human body identity information determination unit is used to determine the identity information of the first human body according to the feature information of the candidate face having the smallest distance value to the second area.
[0175] In one embodiment of the present application, the behavior detection device further includes: a seventh acquisition unit and a target object determination unit.
[0176] The seventh acquisition unit is used to acquire a second thermal image collected by the thermal imaging camera device within a preset time period.
[0177] The target object determination unit is used to determine that a preset target object exists among the target objects detected by the second thermal image if there is a third area in the second thermal image whose temperature is greater than a second preset temperature threshold and less than a first preset temperature threshold.
[0178] In one embodiment of the present application, the behavior detection device further includes: an alarm unit.
[0179] The alarm unit is used to output an alarm control instruction to a preset alarm device, and the alarm control instruction is used to instruct the alarm device to sound an alarm.
[0180] As can be seen from the above, the behavior detection device provided by the present application obtains the target human body image collected by the visible light camera device; the target human body image is input into the pre-constructed behavior detection model for processing to obtain the behavior probability matrix corresponding to the target human body image; wherein the value of each element in the behavior probability matrix is used to represent the probability that the target human body in the target human body image has the preset violation behavior corresponding to the element, and the behavior detection model is obtained by training the pre-constructed first deep learning model based on the first preset sample set; each sample data in the first preset sample set includes a sample human body image and a behavior probability matrix corresponding to the sample human body image; if the value of the first element in the behavior probability matrix is greater than the preset probability threshold corresponding to the first element, it is determined that the target human body has the preset violation behavior corresponding to the first element. The above-mentioned behavior detection method only needs to input the target human body image collected by the visible light camera device into the pre-constructed behavior detection model to obtain the behavior probability matrix corresponding to the target human body image, and then it can be determined whether the target human body has a violation behavior or the type of violation behavior according to the value of each element in the behavior probability matrix and the preset probability threshold corresponding to each element. Compared with the prior art, the present application does not need to compare the target human body image with a large number of human body images with human violations, thereby shortening the behavior detection time and improving the efficiency of behavior detection.
[0181] It should be noted that the information interaction, execution process and other contents between the above-mentioned units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the above-mentioned method embodiment part and will not be repeated here.
[0182] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0183] Fig.11 This is a schematic diagram of the structure of a behavior detection device provided in one embodiment of the present application. Fig.11 As shown, the behavior detection device 7 of this embodiment includes: at least one processor 70 ( Fig.11 Only one is shown in the figure) a processor, a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, and when the processor 70 executes the computer program 72, the steps in any one of the above-mentioned behavior detection method embodiments are implemented.
[0184] The behavior detection device 7 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The behavior detection device 7 may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Figure 7 It is only an example of the behavior detection device 7 and does not constitute a limitation on the behavior detection device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.
[0185] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0186] In some embodiments, the memory 71 may be an internal storage unit of the behavior detection device 7, such as a hard disk or memory of the behavior detection device 7. In other embodiments, the memory 71 may also be an external storage device of the behavior detection device 7, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the behavior detection device 7. Further, the memory 71 may also include both an internal storage unit and an external storage device of the behavior detection device 7. The memory 71 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 71 may also be used to temporarily store data that has been output or is to be output.
[0187] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any one of the above-mentioned behavior detection method embodiments can be implemented.
[0188] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in any of the above-mentioned behavior detection method embodiments when executing the computer program product.
[0189] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the behavior detection device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0190] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0191] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0192] In the embodiments provided in the present application, it should be understood that the disclosed devices / network devices and methods can be implemented in other ways. For example, the above-described behavior detection device embodiments are merely schematic. For example, the division of the modules or units is merely a logical function division. There may be other division methods in actual implementation, such as multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.
[0193] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0194] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A behavior detection method, characterized in that: include: Acquire a target human body image captured by a visible light camera device; The target human body image is input into a pre-constructed behavior detection model for processing to obtain a behavior probability matrix corresponding to the target human body image; wherein the value of each element in the behavior probability matrix is used to represent the probability that the target human body in the target human body image has a preset violation behavior corresponding to the element, and the behavior detection model is obtained by training a pre-constructed first deep learning model based on a first preset sample set; each sample data in the first preset sample set includes a sample human body image and a behavior probability matrix corresponding to the sample human body image; If the value of the first element in the behavior probability matrix is greater than the preset probability threshold corresponding to the first element, it is determined that the target human body has the preset violation behavior corresponding to the first element; Acquire a first thermal image captured by a thermal imaging camera; If there is a first area in the first thermal image whose temperature value is greater than a first preset temperature threshold, determining that a first human body detected by the first thermal image has a first preset violation, and determining coordinate information of the first area; Acquire a first human body image captured by a target visible light camera device associated with the thermal imaging camera device; wherein the target visible light camera device and the thermal imaging camera device are arranged at the same position, and the acquisition time of the first human body image is the same as the acquisition time of the first thermal image; Determining, according to the coordinate information of the first area, coordinate information of a second area in the first human body image corresponding to the first area; Performing face detection on the first human image, and determining coordinate information of each candidate human face contained in the first human image; Determine, according to the coordinate information of each candidate face and the coordinate information of the second region, a distance value between each candidate face and the second region, wherein the distance value is a straight-line distance value between a midpoint of a rectangular region occupied by the candidate face in the second coordinate system and a midpoint of the second region; The identity information of the first human body is determined according to the feature information of the candidate human face having the smallest distance value to the second area.
2. The behavior detection method according to claim 1, characterized in that: The step of acquiring the target human body image captured by the visible light camera device comprises: Acquire multiple frames of human body images captured by the visible light camera device; Inputting the multiple frames of human body images into a preset face detection model to obtain face score values corresponding to each of the multiple frames of human body images; wherein the face score values are used to indicate the integrity of the faces in the human body images; The human body image with the largest face score value is determined as the target human body image.
3. The behavior detection method according to claim 2, characterized in that: Before acquiring the target human body image captured by the visible light camera device, the behavior detection method further includes: Acquire multiple frames of video images captured by the visible light camera device; Inputting the multiple frames of video images into a preset target detection model for target recognition, and obtaining target recognition results corresponding to each of the multiple frames of video images; The video image including a human body in the target recognition result is determined as the human body image.
4. The behavior detection method according to claim 1, characterized in that: After determining that the target human body has a preset illegal behavior corresponding to the first element, the behavior detection method further includes: Acquire feature information of a human face contained in the target human image; The identity information of the target human body is determined according to the feature information of the human face contained in the target human body image and the correspondence between the feature information of each preset human face and the identity information stored in advance.
5. The behavior detection method according to claim 1, characterized in that: Also includes: Acquire a second thermal image captured by the thermal imaging camera device within a preset time period; If there is a third area in the second thermal image whose temperature is greater than the second preset temperature threshold and less than the first preset temperature threshold, it is determined that there is a preset target object in the target objects detected by the second thermal image.
6. The behavior detection method according to any one of claims 1 to 5, characterized in that: After determining that the target human body has a preset illegal behavior corresponding to the first element, the behavior detection method further includes: Output an alarm control instruction to a preset alarm device, wherein the alarm control instruction is used to instruct the alarm device to sound an alarm.
7. A behavior detection device, characterized in that: include: A first acquisition unit, used to acquire a target human body image captured by a visible light camera device; A behavior detection unit is used to input the target human body image into a pre-constructed behavior detection model for processing to obtain a behavior probability matrix corresponding to the target human body image; wherein the value of each element in the behavior probability matrix is used to represent the probability that the target human body in the target human body image has a preset violation behavior corresponding to the element, and the behavior detection model is obtained by training a pre-constructed first deep learning model based on a first preset sample set; each sample data in the first preset sample set includes a sample human body image and a behavior probability matrix corresponding to the sample human body image; a behavior determination unit, configured to determine that the target human body has a preset violation behavior corresponding to the first element if the value of the first element in the behavior probability matrix is greater than a preset probability threshold corresponding to the first element; The fifth acquisition unit is used to acquire a first thermal image collected by the thermal imaging camera device; The violation determination unit is used to determine that a first human body detected by the first thermal image has a first preset violation if there is a first area with a temperature value greater than a first preset temperature threshold in the first thermal image, and determine the coordinate information of the first area; The sixth acquisition unit is used to acquire a first human body image acquired by a target visible light camera device associated with the thermal imaging camera device; wherein the target visible light camera device and the thermal imaging camera device are arranged at the same position, and the acquisition time of the first human body image is the same as the acquisition time of the first thermal image; The first coordinate information determining unit is used to determine the coordinate information of a second area corresponding to the first area in the first human body image according to the coordinate information of the first area; The second coordinate information determining unit is used to perform face detection on the first human image and determine the coordinate information of each candidate human face contained in the first human image; The distance value determination unit is used to determine the distance value between each candidate face and the second area according to the coordinate information of each candidate face and the coordinate information of the second area, wherein the distance value is the straight-line distance value between the midpoint of the rectangular area occupied by the candidate face in the second coordinate system and the midpoint of the second area; The first human body identity information determination unit is used to determine the identity information of the first human body according to the feature information of the candidate face having the smallest distance value to the second area.
8. A behavior detection device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Behavior identification method and device
CN107247946A
Method and system for detecting invasion of small animals in a cabinet based on a low-cost infrared thermal vision technology
CN109657737A
Illegal event detection method and device, electronic equipment and storage medium
CN111310665A
Alarm method and device, storage medium and electronic device
CN111507268A
