Dangerous behavior recognition method and system based on super-resolution reconstruction and related device

Abstract

The application discloses a dangerous behavior recognition method and system based on super-resolution reconstruction and related equipment, and the method comprises the following steps: acquiring a plurality of continuous frames of to-be-recognized images; performing super-resolution reconstruction on the plurality of continuous frames of to-be-recognized images according to a trained super-resolution reconstruction model to obtain a plurality of continuous frames of super-resolution images, wherein the resolution of the super-resolution images is higher than that of the to-be-recognized images; performing dangerous behavior recognition on the super-resolution images by using a trained behavior recognition model to obtain a target behavior category corresponding to the to-be-recognized images, wherein the target behavior category comprises one or more of a plurality of preset behavior categories, and the plurality of preset behavior categories comprise no danger, abnormal protective equipment and climbing. Compared with the prior art, the application is beneficial to improving the accuracy of dangerous behavior recognition.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system and related equipment for identifying dangerous behaviors based on super-resolution reconstruction. Background Technology

[0002] With the development of science and technology, especially the development of image acquisition equipment such as cameras, solutions for monitoring dangerous behaviors of places or objects that need to be monitored (such as processing personnel and equipment operators) based on images acquired by cameras are receiving increasing attention.

[0003] In existing technologies, dangerous behavior is typically determined by analyzing a single frame captured by a surveillance camera. The problem with this approach is that surveillance cameras usually have low resolution, resulting in low-resolution images. This makes it difficult to clearly identify dangerous behaviors when analyzing these images, hindering the accuracy of dangerous behavior identification.

[0004] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0005] The main objective of this invention is to provide a method, system, and related equipment for identifying dangerous behaviors based on super-resolution reconstruction, aiming to solve the problem that the existing approach of directly analyzing a single frame of image captured by a surveillance camera to determine the existence of dangerous behaviors is not conducive to improving the accuracy of dangerous behavior identification.

[0006] To achieve the above objectives, a first aspect of the present invention provides a method for identifying dangerous behaviors based on super-resolution reconstruction, wherein the method includes:

[0007] Acquire multiple consecutive frames of images to be recognized;

[0008] The super-resolution reconstruction model is used to perform super-resolution reconstruction on the above-mentioned consecutive frames of images to be identified to obtain consecutive super-resolution images, wherein the resolution of the super-resolution images is higher than that of the images to be identified.

[0009] Based on the super-resolution image, dangerous behavior is identified using a trained behavior recognition model to obtain the target behavior category corresponding to the image to be identified. The target behavior category includes one or more of a variety of preset behavior categories, including no danger, abnormal protective gear, and climbing.

[0010] Optionally, the above-mentioned super-resolution reconstruction of the above-mentioned consecutive multi-frame images to be identified based on the trained super-resolution reconstruction model to obtain consecutive multi-frame super-resolution images includes:

[0011] Each frame of the above-mentioned image to be identified is sequentially input into the above-mentioned trained super-resolution reconstruction model. The super-resolution reconstruction model is used to perform super-resolution reconstruction on each of the above-mentioned images to be identified to obtain a frame of super-resolution image corresponding to each of the above-mentioned images to be identified.

[0012] The image to be identified is obtained by taking a picture of the target area with a camera, and the super-resolution reconstruction model uses interpolation to perform super-resolution reconstruction.

[0013] Optionally, the super-resolution reconstruction model described above is pre-trained according to the following steps:

[0014] The low-resolution training image in the first training data is input into the super-resolution reconstruction model. Pixel interpolation is performed according to the super-resolution reconstruction model to generate a high-resolution reconstructed image corresponding to the low-resolution training image. The first training data includes multiple sets of first training image groups. Each set of the first training image groups includes a low-resolution training image and a high-resolution training image. One low-resolution training image corresponds to one high-resolution training image, and the resolution of the low-resolution training image is lower than that of the high-resolution training image.

[0015] Based on the high-resolution training image corresponding to the low-resolution training image and the high-resolution reconstructed image corresponding to the low-resolution training image, the model parameters of the super-resolution reconstruction model are adjusted, and the step of inputting the low-resolution training image in the first training data into the super-resolution reconstruction model is continued until the first preset training condition is met, so as to obtain the trained super-resolution reconstruction model.

[0016] Optionally, the high-resolution training image is an image acquired by a high-definition camera, and the low-resolution training image corresponding to the high-resolution training image is an image acquired by a low-definition camera capturing the shooting area corresponding to the high-resolution training image. The resolution of the high-definition camera is higher than that of the low-definition camera.

[0017] Optionally, the high-resolution training image is an image acquired by a high-definition camera, and the low-resolution training image corresponding to the high-resolution training image is an image obtained by processing the high-resolution training image based on a preset interference method, which includes at least one of applying jitter and blurring.

[0018] Optionally, based on the super-resolution image, the above-mentioned dangerous behavior identification is performed using a trained behavior recognition model to obtain the target behavior category corresponding to the image to be identified, including:

[0019] Input the above-mentioned consecutive multi-frame super-resolution images into the above-trained behavior recognition model;

[0020] In the above-trained behavior recognition model, a region weight value is set for each image sub-region of each frame of super-resolution image. The region weight value of the same image sub-region corresponding to each frame of super-resolution image in the above-mentioned consecutive frames of super-resolution image is the same. The region weight value of the image sub-region is determined according to the amount of change of pixel value of the image sub-region in each frame of super-resolution image. The region weight value corresponding to the image sub-region with a large amount of change of pixel value is greater than the region weight value corresponding to the image sub-region with a small amount of change of pixel value.

[0021] In the trained behavior recognition model, feature vectors are obtained by performing feature recognition on the super-resolution images of the above-mentioned consecutive frames based on the above-mentioned region weight values, and dangerous behavior classification is performed on the above-mentioned feature vectors to obtain the above-mentioned target behavior category.

[0022] Optionally, the above behavior recognition model is pre-trained according to the following steps:

[0023] The continuous multi-frame training super-resolution images in the second training data are input into the behavior recognition model. The behavior recognition model sets training region weight values ​​for each training image sub-region of each frame of training super-resolution images. The behavior recognition model performs feature recognition on the continuous multi-frame training super-resolution images according to the training region weight values ​​to obtain training feature vectors. The training feature vectors are then used to classify dangerous behaviors to obtain the training target behavior category corresponding to the training super-resolution images. The second training data includes multiple sets of second training images. Each set of second training images includes a set of continuous multi-frame training super-resolution images and their corresponding training actual dangerous behavior categories.

[0024] Based on the training target behavior category corresponding to the above-mentioned consecutive multi-frame training super-resolution images and the training actual dangerous behavior category corresponding to the above-mentioned consecutive multi-frame images to be identified, the model parameters of the above-mentioned behavior recognition model are adjusted, and the above-mentioned step of inputting the consecutive multi-frame training super-resolution images in the second training data into the above-mentioned behavior recognition model is continued until the preset second training condition is met, so as to obtain the trained behavior recognition model.

[0025] Optionally, after identifying dangerous behaviors based on the super-resolution image using a trained behavior recognition model to obtain the target behavior category corresponding to the image to be identified, the method further includes:

[0026] Each behavior in the above target behavior category is matched with a preset dangerous behavior level table and all corresponding dangerous levels are obtained. The matching dangerous level with the highest dangerous level is taken as the target dangerous level corresponding to the above target behavior category.

[0027] A hazard warning will be issued based on the aforementioned target hazard level.

[0028] A second aspect of the present invention provides a dangerous behavior recognition system based on super-resolution reconstruction, wherein the dangerous behavior recognition system based on super-resolution reconstruction includes:

[0029] The image acquisition module is used to acquire multiple consecutive frames of images to be recognized.

[0030] The super-resolution reconstruction module is used to perform super-resolution reconstruction on the above-mentioned consecutive multi-frame images to be identified based on the trained super-resolution reconstruction model to obtain consecutive multi-frame super-resolution images, wherein the resolution of the super-resolution images is higher than that of the images to be identified.

[0031] The dangerous behavior recognition module is used to identify dangerous behaviors based on the super-resolution image and through a trained behavior recognition model to obtain the target behavior category corresponding to the image to be recognized. The target behavior category includes one or more of a variety of preset behavior categories, including no danger, abnormal protective gear, and climbing.

[0032] A third aspect of the present invention provides a smart terminal, the smart terminal including a memory, a processor, and a dangerous behavior recognition program based on super-resolution reconstruction stored in the memory and executable on the processor, wherein when the dangerous behavior recognition program based on super-resolution reconstruction is executed by the processor, it implements the steps of any of the dangerous behavior recognition methods based on super-resolution reconstruction.

[0033] As can be seen from the above, in the present invention, multiple consecutive frames of images to be identified are acquired; super-resolution reconstruction of the multiple consecutive frames of images to be identified is performed on the trained super-resolution reconstruction model to obtain multiple consecutive super-resolution images, wherein the resolution of the super-resolution images is higher than that of the images to be identified; based on the super-resolution images, dangerous behavior is identified through a trained behavior recognition model to obtain the target behavior category corresponding to the images to be identified, wherein the target behavior category includes one or more of a variety of preset behavior categories, including no danger, abnormal protective gear, and climbing.

[0034] Compared to existing technologies, the present invention does not directly identify dangerous behaviors based on a low-resolution image after acquiring it. Instead, it performs super-resolution reconstruction on the image to be identified, obtaining a higher-resolution super-resolution image. This higher-resolution super-resolution image is then used to identify dangerous behaviors, thus improving the accuracy of dangerous behavior identification. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating a dangerous behavior identification method based on super-resolution reconstruction provided in an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of a dangerous behavior recognition system based on super-resolution reconstruction provided in an embodiment of the present invention;

[0038] Figure 3 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. Detailed Implementation

[0039] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0040] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0041] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0042] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0043] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to classification." Similarly, the phrases "if determined" or "if classified to [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once classified to [the described condition or event]," or "in response to classification to [the described condition or event]."

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0046] With the development of science and technology, especially the development of image acquisition equipment such as cameras, more and more surveillance cameras are being set up in various areas. Solutions for monitoring dangerous behaviors of places that need to be monitored (such as construction sites) or objects that need to be monitored (such as workers, processing personnel, and equipment operators) based on images captured by cameras are receiving increasing attention.

[0047] In existing technologies, a single frame of image captured by a surveillance camera is typically analyzed to determine the presence of dangerous behavior. The problem with this technology is that surveillance cameras usually have low resolution, resulting in low-resolution images. This makes it difficult to clearly identify dangerous behaviors when analyzing these images, hindering the accuracy of dangerous behavior identification. For example, the data quality (especially resolution) acquired by image acquisition devices such as surveillance cameras installed at construction sites is generally low, making accurate identification of dangerous behaviors difficult.

[0048] Meanwhile, existing technologies typically only allow for the identification of dangerous behaviors based on single-frame images. However, the behavior of the target object (such as a worker who needs to be monitored) is usually continuous, making it difficult to accurately judge from a single-frame image. Therefore, this is not conducive to improving the accuracy of dangerous behavior identification.

[0049] Furthermore, existing technologies typically require monitoring personnel to manually identify the images, which consumes a lot of human resources (e.g., multiple monitoring teams need to be on duty 24 hours a day), and the identification efficiency is not high, which is not conducive to improving the efficiency of identifying dangerous behaviors.

[0050] To address at least one of the aforementioned problems, the present invention involves: acquiring multiple consecutive frames of images to be identified; performing super-resolution reconstruction on the multiple consecutive frames of images to be identified using a trained super-resolution reconstruction model to obtain multiple consecutive super-resolution images, wherein the resolution of the super-resolution images is higher than that of the images to be identified; and identifying dangerous behaviors using a trained behavior recognition model based on the super-resolution images to obtain the target behavior category corresponding to the images to be identified, wherein the target behavior category includes one or more of a plurality of preset behavior categories, including no danger, abnormal protective gear, and climbing.

[0051] Compared to existing technologies, the present invention does not directly identify dangerous behaviors based on a low-resolution image after acquiring it. Instead, it performs super-resolution reconstruction on the image to be identified, obtaining a higher-resolution super-resolution image. This higher-resolution super-resolution image is then used to identify dangerous behaviors, thus improving the accuracy of dangerous behavior identification.

[0052] Furthermore, the present invention can combine multiple consecutive frames of images for dangerous behavior identification, comprehensively considering continuous behaviors and further improving the accuracy of dangerous behavior identification. Additionally, based on the present invention, image super-resolution reconstruction can be automatically performed using a trained super-resolution reconstruction model, and dangerous behavior identification can be automatically performed using a trained behavior recognition model, eliminating the need for manual processing and identification by monitoring personnel, thus improving the efficiency of dangerous behavior identification.

[0053] Exemplary methods

[0054] like Figure 1 As shown, this embodiment of the invention provides a dangerous behavior identification method based on super-resolution reconstruction. Specifically, the method includes the following steps:

[0055] Step S100: Acquire multiple consecutive frames of images to be recognized.

[0056] The aforementioned images to be identified are obtained by capturing images of the target area and / or the target object to be monitored by a pre-set surveillance camera. The surveillance camera captures multiple frames of images to be identified continuously, thereby determining whether the target object exhibits dangerous behavior based on the continuous images to be identified.

[0057] In this embodiment, the target area is a construction site, and the target objects are the workers at the construction site. In actual use, the target area and the corresponding target objects can be determined according to actual needs. For example, the target area could also be a railway inspection area, and the target objects could also be inspection personnel; no specific limitations are made here.

[0058] Specifically, the number of frames of the image to be identified can be preset (e.g., preset to 8 frames) or adjusted according to actual needs. In this embodiment, the number of frames acquired is 8 to 16, and there is no specific limitation.

[0059] Step S200: Perform super-resolution reconstruction on the above-mentioned consecutive multi-frame images to be identified according to the trained super-resolution reconstruction model to obtain consecutive multi-frame super-resolution images, wherein the resolution of the super-resolution images is higher than that of the images to be identified.

[0060] The aforementioned trained super-resolution reconstruction model is a pre-trained model used to perform super-resolution reconstruction of images to improve image resolution. In this embodiment, the super-resolution reconstruction model is a deep reinforcement learning (DRL) model, but this is not a specific limitation.

[0061] In this embodiment, the super-resolution reconstruction model sequentially performs super-resolution reconstruction on each frame of the image to be identified to improve its image resolution, thereby enabling better feature recognition. In this embodiment, the above-mentioned super-resolution reconstruction of the aforementioned consecutive frames of the image to be identified based on the trained super-resolution reconstruction model to obtain consecutive frames of super-resolution images includes:

[0062] Each frame of the image to be identified is sequentially input into the trained super-resolution reconstruction model. The super-resolution reconstruction model performs super-resolution reconstruction on each of the images to be identified to obtain a super-resolution image corresponding to each image to be identified. The images to be identified are obtained by taking pictures of the target area with a camera, and the super-resolution reconstruction model uses interpolation to perform super-resolution reconstruction.

[0063] Furthermore, the super-resolution reconstruction model is pre-trained according to the following steps: inputting the low-resolution training image in the first training data into the super-resolution reconstruction model, and performing pixel interpolation according to the super-resolution reconstruction model to generate a high-resolution reconstructed image corresponding to the low-resolution training image. The first training data includes multiple sets of first training image groups, each set of the first training image groups includes a low-resolution training image and a high-resolution training image, one low-resolution training image corresponds to one high-resolution training image, and the resolution of the low-resolution training image is lower than that of the high-resolution training image.

[0064] Based on the high-resolution training image corresponding to the low-resolution training image and the high-resolution reconstructed image corresponding to the low-resolution training image, the model parameters of the super-resolution reconstruction model are adjusted, and the step of inputting the low-resolution training image in the first training data into the super-resolution reconstruction model is continued until the first preset training condition is met, so as to obtain the trained super-resolution reconstruction model.

[0065] The model parameters of the super-resolution reconstruction model are used to determine whether interpolation is used and, specifically, which interpolation method (e.g., average pooling, max pooling). The first preset training condition includes that the number of iterations corresponding to the super-resolution reconstruction model is not less than a preset first iteration threshold, or that the loss value between the high-resolution training image and the high-resolution reconstructed image is not less than a preset first loss threshold. Other conditions may also be included, which are not specifically limited here.

[0066] It should be noted that the high-resolution training images and the low-resolution training images mentioned above contain the same image content. For example, they are images with different resolutions acquired from the same region or object.

[0067] In one application scenario, the aforementioned high-resolution training image is an image acquired by a high-definition camera, and the corresponding low-resolution training image is an image acquired by a low-definition camera capturing the area corresponding to the high-resolution training image. The resolution of the high-definition camera is higher than that of the low-definition camera.

[0068] In another application scenario, the high-resolution training image is an image acquired by a high-definition camera, and the low-resolution training image corresponding to the high-resolution training image is an image obtained by processing the high-resolution training image based on a preset interference method, which includes at least one of applying jitter and blurring.

[0069] In this embodiment, the training images in the first training data are also obtained by capturing images of the target area that needs to be identified for dangerous behavior. Specifically, a high-resolution training image is obtained by pre-capturing images of the target area using a high-definition camera. Then, methods such as jittering and blurring are applied to the high-resolution training image to reduce its resolution and obtain the corresponding low-resolution training image. During the training of the super-resolution reconstruction model (i.e., a deep reinforcement learning model), the current environment perceived by the deep reinforcement learning model is the pixel values ​​of all pixels in the entire image. Each pixel acts as a reinforcement learning agent, and its behavior is defined as interpolating with its nearest neighbor pixels (each pixel has 8 neighboring points). This behavior can be used to restore the image interference factors such as jitter. Whether to perform interpolation and what interpolation method to use (average pooling, max pooling, etc.) is a policy that the agent needs to learn to decide (i.e., the policy in reinforcement learning). The model parameters can be used to limit whether interpolation is needed for each pixel and what interpolation method is used. Furthermore, according to the definition of super-resolution reconstruction, after all agents of the deep reinforcement learning model have completed their actions, the obtained high-resolution reconstructed image is compared with the original high-resolution training image. The features of the two images are extracted by a convolutional network (CNN), the cosine distance is calculated, and the difference in distance is used as the loss. The model parameters are updated through backpropagation.

[0070] It should be noted that the aforementioned super-resolution reconstruction model corresponds to a reconstruction scale, which is used to limit the resolution enhancement factor during image reconstruction. The reconstruction scale can be input by the user in real time or a fixed value can be preset. In this embodiment, a fixed value (e.g., set to 4 times) is used as an example, that is, a fixed reconstruction scale is used both in the use of the super-resolution reconstruction model and during the training process.

[0071] When the reconstruction scale is input by the user, the user can determine its value in real time according to actual needs and input it into the model to limit the resolution improvement factor. It should be noted that when the reconstruction scale is input by the user, multiple reconstruction scales are also pre-set during the training of the super-resolution reconstruction model, and corresponding low-resolution training images and high-resolution training images are provided for different reconstruction scales. It should be noted that the reconstruction scale input by the user is one of several pre-set reconstruction scales; that is, the user selects from a range of pre-set reconstruction scales.

[0072] In one application scenario, weight values ​​can be set for different regions of the image to be identified, so that the super-resolution reconstruction model pays more attention to the regions that need attention, thereby improving the image reconstruction efficiency of the super-resolution reconstruction model. Specifically, the weight values ​​of different regions in the image to be identified can be preset by the user according to actual needs (for example, setting a higher weight for the center of the image, or a location where the target object frequently appears, or a location where important equipment is located, or a more dangerous area), or adjusted by the user in real time (the user determines which region needs to be given a higher weight based on the current image to be identified), or determined according to the complexity of the pixel values ​​in the current frame of the image to be identified (regions with large pixel values ​​or large variance of pixel values ​​have higher weight values), or determined according to the degree of change of pixel values ​​in the same region of the image to be identified in consecutive frames (or all frames), with regions with greater changes in pixel values ​​having higher weight values.

[0073] Step S300: Based on the super-resolution image, dangerous behavior is identified using a trained behavior recognition model to obtain the target behavior category corresponding to the image to be identified. The target behavior category includes one or more of a variety of preset behavior categories, including no danger, abnormal protective gear, and climbing.

[0074] The aforementioned trained behavior recognition model is a pre-trained model used to extract features from images (or consecutive image frames) and identify behavior categories within them. The target behavior category refers to the identified irregular (potentially dangerous) behaviors that may exist in the original, consecutive multi-frame images to be recognized. In one application scenario, the behavior recognition model can identify dangerous behaviors in a single frame of an image, and then combine the dangerous behaviors identified for each frame of the image to be recognized as the target behavior category corresponding to the image to be recognized.

[0075] In this embodiment, dangerous behavior recognition is performed by combining the trained behavior recognition model with multiple consecutive frames of super-resolution images to better identify dangerous behaviors by combining the actions of the target object between consecutive frames. Furthermore, this embodiment can also set weight values ​​for each region of the super-resolution image. It should be noted that when setting weight values ​​for each region of the super-resolution image, the method can be similar to setting weight values ​​for different regions of the image to be recognized. For example, the user can pre-set the weight according to actual needs (e.g., setting a higher weight for the image center, the location where the target object frequently appears, the location of important equipment, or more dangerous areas), or the user can adjust it in real time (the user confirms which region needs a higher weight based on the current super-resolution image). The weight can also be determined based on the complexity of the pixel values ​​in the current frame of the super-resolution image (regions with large pixel values ​​or large variance corresponding to pixel values ​​have higher weight values), or based on the degree of change in pixel values ​​in the same region of consecutive frames (or all frames) of the super-resolution image. Regions with greater pixel value changes have higher weight values, but no specific limitation is made here.

[0076] In this embodiment, the above-mentioned dangerous behavior identification based on the super-resolution image and the trained behavior recognition model to obtain the target behavior category corresponding to the image to be identified includes:

[0077] Input the above-mentioned consecutive multi-frame super-resolution images into the above-trained behavior recognition model;

[0078] In the above-trained behavior recognition model, a region weight value is set for each image sub-region of each frame of super-resolution image. The region weight value of the same image sub-region corresponding to each frame of super-resolution image in the above-mentioned consecutive frames of super-resolution image is the same. The region weight value of the image sub-region is determined according to the amount of change of pixel value of the image sub-region in each frame of super-resolution image. The region weight value corresponding to the image sub-region with a large amount of change of pixel value is greater than the region weight value corresponding to the image sub-region with a small amount of change of pixel value.

[0079] In the trained behavior recognition model, feature vectors are obtained by performing feature recognition on the super-resolution images of the above-mentioned consecutive frames based on the above-mentioned region weight values, and dangerous behavior classification is performed on the above-mentioned feature vectors to obtain the above-mentioned target behavior category.

[0080] In this embodiment, all super-resolution images have the same image size and resolution, so identical sub-regions can be obtained during image sub-region division. For example, dividing each image equally according to a preset number of regions can yield corresponding sub-regions. In this embodiment, the weight value of each sub-region is determined based on the change in pixel values ​​across multiple consecutive frames corresponding to that region. This change in pixel values ​​can be the difference, variance, or standard deviation of pixel values ​​across multiple consecutive frames. For example, for each pixel in a sub-region, its variance is determined based on multiple consecutive frames, and the sum of the variances of all pixels in the sub-region can be used as the change in pixel values ​​for that region. When setting weight values, the weight value of the sub-region with a larger change in pixel values ​​is greater than that of the sub-region with a smaller change in pixel values. For example, the corresponding weight values ​​can be set to 1, 2, 3, etc., sequentially according to the magnitude of the change in pixel values. Alternatively, the specific weight values ​​can be fine-tuned or standardized to the 0-1 range to improve computational efficiency and achieve better results.

[0081] In this embodiment, the behavior recognition model is pre-trained according to the following steps:

[0082] The continuous multi-frame training super-resolution images in the second training data are input into the behavior recognition model. The behavior recognition model sets training region weight values ​​for each training image sub-region of each frame of training super-resolution images. The behavior recognition model performs feature recognition on the continuous multi-frame training super-resolution images according to the training region weight values ​​to obtain training feature vectors. The training feature vectors are then used to classify dangerous behaviors to obtain the training target behavior category corresponding to the training super-resolution images. The second training data includes multiple sets of second training images. Each set of second training images includes a set of continuous multi-frame training super-resolution images and their corresponding training actual dangerous behavior categories.

[0083] Based on the training target behavior category corresponding to the above-mentioned consecutive multi-frame training super-resolution images and the training actual dangerous behavior category corresponding to the above-mentioned consecutive multi-frame images to be identified, the model parameters of the above-mentioned behavior recognition model are adjusted, and the above-mentioned step of inputting the consecutive multi-frame training super-resolution images in the second training data into the above-mentioned behavior recognition model is continued until the preset second training condition is met, so as to obtain the trained behavior recognition model.

[0084] The second training data mentioned above is pre-collected data used to train the behavior recognition model. It includes multiple sets of consecutive multi-frame training super-resolution images, each set of which has pre-labeled training actual dangerous behavior categories. The corresponding training target behavior category is the dangerous category that the model might contain in the image after feature recognition and classification matching during model training. The second preset training condition includes that the number of iterations for the behavior recognition model is not less than a preset second iteration threshold, or that the loss value between the training target behavior category and the training actual dangerous behavior category is not less than a preset second loss threshold. Other conditions may also be included, which are not specifically limited here.

[0085] In this embodiment, multiple consecutive frames (e.g., 8-16 frames) of high-resolution images of the target area are pre-acquired (the resolution of which is the same as the resolution of the super-resolution images used in the model) as training super-resolution images. Alternatively, corresponding training super-resolution images can be obtained by super-resolution reconstruction of the acquired low-resolution images. These images are then manually labeled with their corresponding training actual dangerous behavior categories, such as no dangerous behavior, or one or more dangerous behaviors such as abnormal protective gear, climbing, or smoking. The corresponding training super-resolution images are then input into the behavior recognition model for dangerous behavior recognition.

[0086] In this embodiment, the behavior recognition model is specifically a gated recurrent neural network (GRU) model. Both the super-resolution reconstruction model and the behavior recognition model can be trained using semi-supervised learning methods, which are not specifically limited here. It should be noted that the data involved in the model's use and training are similar or corresponding; for example, the training super-resolution image corresponds to the super-resolution image, and the training feature vector has a corresponding feature vector. The distinction between data involved in the model training and use processes is made only by name, but this is not considered a specific limitation.

[0087] Specifically, in this embodiment, a super-resolution reconstruction model reads low-resolution images of the engineering site and reconstructs them into high-resolution training super-resolution images. These training super-resolution images are easily used for risk behavior identification. These training super-resolution images are input into the behavior recognition model, where the behavior recognition network processes them to obtain a 1D vector (i.e., a training feature vector). The model's softmax layer outputs the probabilities of different risk behaviors. The risk behavior with the highest probability (including risk-free behaviors) is selected and compared with the true label (i.e., the training actual dangerous behavior category). The GRU model parameters are updated based on cross-entropy backpropagation. Thus, based on this embodiment, the scheme can achieve accurate risk identification of low-resolution engineering videos by combining DRL and GRU.

[0088] Furthermore, after identifying dangerous behaviors based on the super-resolution image using a trained behavior recognition model and obtaining the target behavior category corresponding to the image to be identified, the method further includes: matching each behavior in the target behavior category with a preset dangerous behavior level table and obtaining all corresponding matching danger levels, taking the matching danger level with the highest danger level as the target danger level corresponding to the target behavior category; and issuing a danger alarm based on the target danger level.

[0089] The aforementioned hazard behavior level table is a pre-set table used to store different hazardous behaviors and their corresponding hazard levels. For example, it can pre-store the hazard level as Level 1 for no hazard, Level 2 for smoking, and Level 3 for abnormal protective gear and climbing. The target hazard level is determined based on the identified target behavior category. It should be noted that when multiple hazardous behaviors are identified, such as abnormal protective gear and smoking, the highest matching hazard level is used as the target hazard level, i.e., Level 3. Then, a hazard alarm is triggered based on the current target hazard level. Specifically, different alarm methods can be pre-set for different hazard levels. For example, Level 1 may not require an alarm, Level 2 may trigger an audible and visual alarm, and Level 3 may trigger an audible and visual alarm while simultaneously sending an alarm message to a pre-set mobile device, thereby improving the safety of the construction site.

[0090] As can be seen from the above, in the present invention, after acquiring a low-resolution image to be identified, dangerous behavior identification is not performed directly based on it. Instead, super-resolution reconstruction is performed on the image to be identified to obtain a high-resolution super-resolution image, thereby improving the accuracy of dangerous behavior identification.

[0091] Exemplary device

[0092] like Figure 2 As shown, corresponding to the above-described dangerous behavior recognition method based on super-resolution reconstruction, this embodiment of the invention also provides a dangerous behavior recognition system based on super-resolution reconstruction, the above-described dangerous behavior recognition system based on super-resolution reconstruction comprising:

[0093] The image acquisition module 410 is used to acquire multiple consecutive frames of images to be recognized.

[0094] The super-resolution reconstruction module 420 is used to perform super-resolution reconstruction on the above-mentioned consecutive multi-frame images to be identified according to the trained super-resolution reconstruction model to obtain consecutive multi-frame super-resolution images, wherein the resolution of the super-resolution images is higher than that of the images to be identified.

[0095] The dangerous behavior recognition module 430 is used to recognize dangerous behaviors based on the super-resolution image and through a trained behavior recognition model to obtain the target behavior category corresponding to the image to be recognized. The target behavior category includes one or more of a variety of preset behavior categories, including no danger, abnormal protective gear, and climbing.

[0096] Specifically, in this embodiment, the specific functions of the above-mentioned dangerous behavior recognition system based on super-resolution reconstruction and its modules can be referred to the corresponding description in the above-mentioned dangerous behavior recognition method based on super-resolution reconstruction, and will not be repeated here.

[0097] It should be noted that the division of the various modules in the above-mentioned dangerous behavior recognition system based on super-resolution reconstruction is not unique and is not intended as a specific limitation.

[0098] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 3 As shown. The aforementioned smart terminal includes a processor and a memory. The memory of the smart terminal includes a dangerous behavior recognition program based on super-resolution reconstruction, and the memory provides an environment for the operation of the dangerous behavior recognition program based on super-resolution reconstruction. When the dangerous behavior recognition program based on super-resolution reconstruction is executed by the processor, it implements the steps of any of the aforementioned dangerous behavior recognition methods based on super-resolution reconstruction. It should be noted that the aforementioned smart terminal may also include other functional modules or units, which are not specifically limited here.

[0099] Those skilled in the art will understand that Figure 3 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the smart terminal to which the present invention is applied. Specifically, the smart terminal may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0100] This invention also provides a computer-readable storage medium storing a dangerous behavior recognition program based on super-resolution reconstruction. When executed by a processor, the dangerous behavior recognition program based on super-resolution reconstruction implements the steps of any dangerous behavior recognition method based on super-resolution reconstruction provided in this invention.

[0101] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0102] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0103] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0104] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0105] In the embodiments provided by this invention, it should be understood that the disclosed systems / smart terminals and methods can be implemented in other ways. For example, the system / smart terminal embodiments described above are merely illustrative. For instance, the division of modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0106] If the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0107] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not mean that the essence of the corresponding technical solutions deviates from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for identifying dangerous behaviors based on super-resolution reconstruction, characterized in that, The method includes: Acquire multiple consecutive frames of images to be recognized; The super-resolution reconstruction model is used to perform super-resolution reconstruction on the consecutive frames of the image to be identified to obtain consecutive super-resolution images, wherein the resolution of the super-resolution images is higher than that of the image to be identified. The consecutive multi-frame super-resolution images are input into a trained behavior recognition model. In the trained behavior recognition model, a region weight value is set for each sub-region of each super-resolution image frame. The region weight value is the same for the same sub-region corresponding to each super-resolution image frame. The region weight value of the sub-region is determined based on the change in pixel values ​​of the sub-region in each super-resolution image frame; the region weight value corresponding to the sub-region with a larger change in pixel values ​​is greater than the region weight value corresponding to the sub-region with a smaller change in pixel values. In the trained behavior recognition model, feature vectors are obtained by performing feature recognition on the consecutive multi-frame super-resolution images based on the region weight values. The feature vectors are then used to classify dangerous behaviors to obtain target behavior categories. The target behavior categories include one or more of a variety of preset behavior categories, including no danger, abnormal protective gear, and climbing. The trained behavior recognition model is a pre-trained model used to extract features from consecutive image frames and identify behavior categories in the image frames. The behavior recognition model is a gated recurrent neural network model. Each behavior in the target behavior category is matched with a preset danger behavior level table to obtain all corresponding danger levels. The matching danger level with the highest danger level is taken as the target danger level corresponding to the target behavior category. A danger alarm is issued based on the target danger level.

2. The dangerous behavior identification method based on super-resolution reconstruction according to claim 1, characterized in that, The step of performing super-resolution reconstruction on the consecutive multi-frame images to be identified based on the trained super-resolution reconstruction model to obtain consecutive multi-frame super-resolution images includes: Each frame of the image to be identified is sequentially input into the trained super-resolution reconstruction model, and the super-resolution reconstruction model is used to perform super-resolution reconstruction on each image to be identified to obtain a frame of super-resolution image corresponding to each image to be identified. The image to be identified is an image obtained by taking a picture of the target area with a camera, and the super-resolution reconstruction model uses an interpolation method to perform super-resolution reconstruction.

3. The dangerous behavior identification method based on super-resolution reconstruction according to claim 2, characterized in that, The super-resolution reconstruction model is pre-trained according to the following steps: The low-resolution training image in the first training data is input into the super-resolution reconstruction model. Pixel interpolation is performed according to the super-resolution reconstruction model to generate a high-resolution reconstructed image corresponding to the low-resolution training image. The first training data includes multiple sets of first training image groups. Each set of first training image groups includes a low-resolution training image and a high-resolution training image. One low-resolution training image corresponds to one high-resolution training image, and the resolution of the low-resolution training image is lower than that of the high-resolution training image. Based on the high-resolution training image corresponding to the low-resolution training image and the high-resolution reconstructed image corresponding to the low-resolution training image, the model parameters of the super-resolution reconstruction model are adjusted, and the step of inputting the low-resolution training image in the first training data into the super-resolution reconstruction model continues until the first preset training condition is met, so as to obtain the trained super-resolution reconstruction model.

4. The dangerous behavior identification method based on super-resolution reconstruction according to claim 3, characterized in that, The high-resolution training image is an image acquired by a high-definition camera, and the corresponding low-resolution training image is an image acquired by a low-definition camera capturing the area corresponding to the high-resolution training image. The resolution of the high-definition camera is higher than that of the low-definition camera.

5. The dangerous behavior identification method based on super-resolution reconstruction according to claim 3, characterized in that, The high-resolution training image is an image acquired by a high-definition camera. The low-resolution training image corresponding to the high-resolution training image is an image obtained by processing the high-resolution training image based on a preset interference method, which includes at least one of applying jitter and blurring.

6. The dangerous behavior identification method based on super-resolution reconstruction according to claim 1, characterized in that, The behavior recognition model is pre-trained according to the following steps: The behavior recognition model inputs consecutive multi-frame training super-resolution images from the second training data. The behavior recognition model sets training region weight values ​​for each training image sub-region of each frame of training super-resolution images. The behavior recognition model performs feature recognition on the consecutive multi-frame training super-resolution images based on the training region weight values ​​to obtain training feature vectors. The training feature vectors are then used to classify dangerous behaviors to obtain the training target behavior category corresponding to the training super-resolution images. The second training data includes multiple sets of second training images, each set of second training images including a set of consecutive multi-frame training super-resolution images and their corresponding training actual dangerous behavior categories. Based on the training target behavior category corresponding to the consecutive multi-frame training super-resolution images and the training actual dangerous behavior category corresponding to the consecutive multi-frame images to be identified, the model parameters of the behavior recognition model are adjusted, and the step of inputting the consecutive multi-frame training super-resolution images in the second training data into the behavior recognition model is continued until the preset second training condition is met to obtain the trained behavior recognition model.

7. A dangerous behavior recognition system based on super-resolution reconstruction, characterized in that, The system includes: The image acquisition module is used to acquire multiple consecutive frames of images to be recognized. The super-resolution reconstruction module is used to perform super-resolution reconstruction on the consecutive multi-frame images to be identified based on the trained super-resolution reconstruction model to obtain consecutive multi-frame super-resolution images, wherein the resolution of the super-resolution images is higher than that of the images to be identified. A dangerous behavior recognition module is used to input the continuous multi-frame super-resolution images into a trained behavior recognition model. In the trained behavior recognition model, a region weight value is set for each image sub-region of each frame of the super-resolution image. The region weight value is the same for the same image sub-region corresponding to each frame of the continuous multi-frame super-resolution image. The region weight value of the image sub-region is determined based on the change in pixel values ​​of the image sub-region in each frame of the super-resolution image. The region weight value corresponding to the image sub-region with a larger change in pixel values ​​is greater than the region weight value corresponding to the image sub-region with a smaller change in pixel values. In the trained behavior recognition model, feature vectors are obtained by performing feature recognition on the continuous multi-frame super-resolution images based on the region weight values. Dangerous behavior classification is then performed on the feature vectors to obtain target behavior categories. The target behavior categories include one or more of a variety of preset behavior categories, including no danger, abnormal protective gear, and climbing. The trained behavior recognition model is a pre-trained model used to extract features from continuous image frames and identify behavior categories in the image frames. The behavior recognition model is a gated recurrent neural network model. The system is also used to: match each behavior in the target behavior category with a preset dangerous behavior level table and obtain all corresponding matching dangerous levels, take the matching dangerous level with the highest dangerous level as the target dangerous level corresponding to the target behavior category; and issue a dangerous alarm based on the target dangerous level.

8. A smart terminal, characterized in that, The smart terminal includes a memory, a processor, and a dangerous behavior recognition program based on super-resolution reconstruction stored in the memory and executable on the processor. When the dangerous behavior recognition program based on super-resolution reconstruction is executed by the processor, it implements the steps of the dangerous behavior recognition method based on super-resolution reconstruction as described in any one of claims 1-6.

Images