Image recognition method, device, system, equipment, medium and product

Abstract

The embodiment of the invention discloses an image recognition method, device, system and equipment, a medium and a product, and the method comprises the steps: obtaining a to-be-recognized image, inputting the to-be-recognized image into a pre-trained first image recognition model, and obtaining a recognition result of a first target object; determining a target area image according to the recognition result of the first target object; inputting the target area image into a pre-trained second image recognition model to obtain a recognition result of a second target object associated with the first target object; wherein the to-be-recognized image comprises an image collected in an aerial work scene. According to the technical scheme of the embodiment of the invention, the problems of high cost and low efficiency of analyzing high-altitude operation through modes such as manual inspection at present are solved, the target object in the image collected in the high-altitude operation scene can be automatically recognized through the pre-trained neural network model, the recognition efficiency is high, the cost is low, and the recognition efficiency is high. And effective analysis on high-altitude operation is realized.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of image recognition technology, and in particular, to an image recognition method, apparatus, system, device, medium and product. Background Art

[0002] At present, the safety issues of aerial work are becoming increasingly prominent. Among them, aerial work has become the focus and difficulty of safety management due to its particularity and danger. When aerial workers perform tasks such as inspection and maintenance, safety accidents are prone to occur due to the complex working environment and high risks. Therefore, it is necessary to analyze the safety status of aerial workers.

[0003] Traditional safety analysis of aerial work mainly relies on manual inspection and on-site monitoring. Manual inspection requires a lot of manpower and material resources, and the inspection frequency and coverage are limited, making it difficult to detect and correct unsafe behaviors in a timely manner. Although on-site monitoring can observe the work site in real time, it is limited by the layout and angle of the monitoring equipment, making it difficult to fully cover all work areas, and the monitoring personnel need to concentrate for a long time, which is prone to fatigue and omissions. Summary of the invention

[0004] The embodiments of the present invention provide an image recognition method, apparatus, system, equipment, medium and product, which can automatically identify target objects in images collected in aerial work scenarios through a pre-trained neural network model, with high recognition efficiency and low cost, and realize effective analysis of aerial work.

[0005] In a first aspect, an embodiment of the present invention provides an image recognition method, the method comprising:

[0006] Acquire an image to be recognized, and input the image to be recognized into a pre-trained first image recognition model to obtain a recognition result of a first target object;

[0007] Determine a target area image according to a recognition result of the first target object;

[0008] Inputting the target area image into a pre-trained second image recognition model to obtain a recognition result of a second target object associated with the first target object;

[0009] Among them, the images to be identified include images collected in high-altitude work scenes, and the first image recognition model is a model based on a one-time target detection algorithm or a mask area convolutional neural network algorithm.

[0010] In a second aspect, an embodiment of the present invention provides an image recognition device, the device comprising:

[0011] A first image recognition module is used to obtain an image to be recognized, and input the image to be recognized into a pre-trained first image recognition model to obtain a recognition result of a first target object;

[0012] A target area determination module, used to determine a target area image according to a recognition result of the first target object;

[0013] A second image recognition module, used for inputting the target area image into a pre-trained second image recognition model to obtain a recognition result of a second target object associated with the first target object;

[0014] Among them, the images to be identified include images collected in high-altitude work scenes, and the first image recognition model is a model based on a one-time target detection algorithm or a mask area convolutional neural network algorithm.

[0015] In a third aspect, an embodiment of the present invention provides an image recognition system, the system comprising:

[0016] UAV, image transmission module and image recognition module;

[0017] The above-mentioned drone is equipped with a camera for collecting images to be identified;

[0018] The above-mentioned image transmission module is used to transmit the image to be recognized collected by the drone to the image recognition module;

[0019] The above-mentioned image recognition module is used to perform image recognition according to the image recognition method provided by any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0021] one or more processors;

[0022] A memory for storing one or more programs;

[0023] When the one or more programs are executed by one or more processors, the one or more processors implement the image recognition method provided by any embodiment of the present invention.

[0024] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an image recognition method as provided in any embodiment of the present invention.

[0025] In a sixth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the image recognition method provided by any embodiment of the present invention.

[0026] The embodiments of the above invention have the following advantages or beneficial effects:

[0027] The embodiment of the present invention obtains an image to be identified and inputs the image to be identified into a pre-trained first image recognition model to obtain a recognition result of a first target object; determines a target area image according to the recognition result of the first target object; inputs the target area image into a pre-trained second image recognition model to obtain a recognition result of a second target object associated with the first target object; wherein the image to be identified includes images collected in aerial work scenes. The technical solution of the embodiment of the present invention solves the problem of high cost and low efficiency in currently analyzing aerial work through manual inspections and other methods. The pre-trained neural network model can be used to automatically identify target objects in images collected in aerial work scenes, with high recognition efficiency and low cost, thereby achieving effective analysis of aerial work. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of an image recognition method provided by an embodiment of the present invention;

[0029] Figure 2 is a flow chart of an image recognition method provided by an embodiment of the present invention;

[0030] Figure 3 is a flow chart of an image recognition method provided by an embodiment of the present invention;

[0031] Figure 4 is a structural schematic diagram of an image recognition device provided by an embodiment of the present invention;

[0032] Figure 5 is a structural schematic diagram of an image recognition system provided by an embodiment of the present invention;

[0033] Figure 6 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0035] Figure 1This is a flowchart of an image recognition method provided by an embodiment of the present invention. This embodiment can be applied to image recognition scenarios, especially to the recognition of target objects in images collected in aerial work scenarios. The method can be executed by an image recognition device, which can be implemented by software and / or hardware and integrated into a computer device with application development capabilities.

[0036] like Figure 1 As shown, the image recognition method of this embodiment includes the following steps:

[0037] S110, obtaining an image to be recognized, and inputting the image to be recognized into a pre-trained first image recognition model to obtain a recognition result of a first target object.

[0038] Among them, the images to be identified include images collected in high-altitude work scenes, and the first image recognition model is a model based on a one-time target detection algorithm or a mask area convolutional neural network algorithm.

[0039] The image to be identified can be an image of a high-altitude work scene collected by a drone. The high-definition camera and real-time image transmission technology carried by the drone can instantly capture the on-site conditions of the high-altitude work. The first target object can be a high-altitude worker. Before the image to be identified is input into the pre-trained first image recognition model, the image to be identified can be pre-processed to enhance the contrast between the first and second target objects in the image and improve the recognition accuracy of the model. The input size can be, for example, 640x640 pixels, which can be adjusted as needed, and this embodiment is not limited.

[0040] The model based on the one-time target detection algorithm can be a YOLOv5 (You Only Look Once v5) model. The YOLOv5 model has efficient target detection capabilities, can effectively identify high-altitude workers, and can maintain a high recognition accuracy even in complex environments. The image to be recognized is input into the pre-trained first image recognition YOLOv5 model. The YOLOv5 model performs model inference based on the features of the first target object learned in the training phase to obtain the recognition result of the first target object, i.e., the position.

[0041] The model based on the one-time target detection algorithm can also be a mask region convolutional neural network model, namely the MaskR-CNN (Mask Region-based Convolutional Neural Network) model. The Mask R-CNN model is a target detection and instance segmentation algorithm based on deep learning. It can not only determine the category and position of the target, but also generate an accurate mask of the target. The image to be identified is input into the pre-trained first image recognition Mask R-CNN model. The Mask R-CNN model performs model inference based on the features of the first target object learned in the training phase to obtain the recognition result of the first target object, namely the position and mask.

[0042] This embodiment significantly shortens the time from image acquisition to recognition by combining the neural network model with the automated target recognition algorithm, realizes near real-time image recognition, and improves the response speed when emergency events occur. In addition, the generalization ability of the deep learning algorithm ensures stable performance under different light and climate conditions, and enhances the adaptability and practicality of image acquisition and recognition in different environments.

[0043] S120: Determine a target area image according to the recognition result of the first target object.

[0044] The image to be identified is segmented according to the recognition result of the first target object, such as the position of the first target object, to obtain a target area image. Specifically, the area where the first target object is located is segmented from the background area. The target area image can be an image within the area of ​​the rectangular bounding box in the recognition result, or it can be an image corresponding to the target area determined according to the mask.

[0045] S130: Input the target area image into a pre-trained second image recognition model to obtain a recognition result of a second target object associated with the first target object.

[0046] After determining the target area image, a second target object smaller than the first target object in the target area image may be further identified. The second target object may be a safety measure worn by aerial workers during aerial work, such as a safety belt or a safety rope.

[0047] The second image recognition model may be a convolutional neural network model (CNN). The convolutional neural network model may include an input layer, a convolution layer, a pooling layer, and a fully connected layer. The convolutional neural network model is used to perform feature extraction and feature mapping on the target area image to obtain the position and category recognition results of the second target object, and the category recognition results include recognition results of presence and absence.

[0048] According to the recognition result of the second target object, it is possible to further analyze whether the operator is wearing a safety rope or a safety belt, and to issue an alarm if the operator is not wearing a safety rope or a safety belt. In this embodiment, the recognition results of the first target object and the second target object can be output in the form of an image, text or other forms.

[0049] Combine the recognition results of the first target object and the associated second target object to determine whether the operator has worn the safety equipment correctly. If it is found that the safety rope or safety belt is not worn as required, an alarm is automatically triggered and the safety manager is notified through the ground control center or mobile device. At the same time, the violation incident is recorded to provide data support for later analysis and safety management. All image recognition records and historical alarms are stored in the database for long-term tracking and analysis. Regular statistical analysis of data is performed to identify safety hazard trends and provide a basis for formulating more effective safety measures.

[0050] The technical solution of this embodiment obtains the image to be identified and inputs the image to be identified into a pre-trained first image recognition model to obtain the recognition result of the first target object; determines the target area image according to the recognition result of the first target object; inputs the target area image into a pre-trained second image recognition model to obtain the recognition result of the second target object associated with the first target object; wherein the image to be identified includes images collected in aerial work scenes. The technical solution of the embodiment of the present invention solves the problem of high cost and low efficiency in the current analysis of aerial work through manual inspections and other methods. The pre-trained neural network model can be used to automatically identify the target objects in the images collected in the aerial work scenes, with high recognition efficiency and low cost, to achieve effective analysis of aerial work.

[0051] Figure 2 A flowchart of an image recognition method provided in an embodiment of the present invention, which belongs to the same inventive concept as the image recognition method in the above embodiment, further describes the process of outputting the recognition result of the first target object when the pre-trained first image recognition model is a model based on a one-time target detection algorithm. The method can be executed by an image recognition device, which can be implemented by software and / or hardware and integrated into a computer device with application development function.

[0052] like Figure 2 As shown, the image recognition method of this embodiment includes the following steps:

[0053] S210: Obtain an image to be recognized.

[0054] When the pre-trained first image recognition model is a model based on a one-time target detection algorithm, that is, a YOLOv5 model, step S220 is performed.

[0055] S220, performing multi-scale feature extraction on the input image to be recognized through a preset deep network and a feature pyramid network of the first image recognition model to obtain a feature image.

[0056] The network structure of the YOLOv5 model may include an input layer, a Backbone layer, a Neck layer, and a Head layer. The input layer may be used to receive an image to be identified, specifically, an image to be identified after preprocessing. The Backbone layer uses the CSPDarknet53 (Cross Stage PartialDarknet53) network as the backbone network, and improves computational efficiency and feature extraction capabilities through the Cross Stage Partial Networks (CSP) of CSPDarknet53.

[0057] The Neck layer uses the FPN (Feature Pyramid Network) structure, which can integrate feature maps of different scales to achieve multi-scale object recognition. SPP (Spatial Pyramid Pooling) and PAN (Path Aggregation Network) modules can also be introduced to enhance feature fusion.

[0058] The head layer can contain three output layers, each of which is responsible for detecting objects of different sizes; each output layer generates a series of bounding boxes, object category probabilities, and object confidence scores. The object categories corresponding to the object category probabilities include the first target object category and the non-first target object category.

[0059] In this embodiment, the preset deep network may be a CSPDarknet53 network structure, and the feature pyramid network is an FPN network. The CSPDarknet53 network and the FPN network are used to perform multi-scale feature extraction on the input image to be identified to obtain a feature image.

[0060] S230. Predict the feature image through the head network of the first image recognition model to obtain a preliminary recognition result of the target object.

[0061] The head network can be a Head layer, and the preliminary recognition results can be potential bounding boxes, object category probabilities, and object confidence scores.

[0062] Specifically, three output layers pass through the Head layer to predict the potential bounding box, object category probability, and object confidence score based on the feature image.

[0063] S240: Perform non-maximum suppression on the preliminary recognition result to obtain a recognition result of the first target object.

[0064] Non-Maximum Suppression (NMS) can be to remove overlapping prediction boxes and retain the prediction boxes that meet the preset correct standard. The preset correct standard can be set according to actual needs. The recognition result of the first target object is the prediction box that meets the preset correct standard. The recognition result of the first target object includes the position of the first target object in the image to be recognized, that is, the prediction box that meets the preset correct standard, and the corresponding confidence score of the prediction box.

[0065] In an optional implementation, the process of training the first image recognition YOLOv5 model may include the following steps A1-A3:

[0066] Step A1: Acquire a target sample image, and mark the area where the first target object is located in the target sample image to obtain a marked image.

[0067] The target sample images may be images of a training set. For example, images containing a first target object and a second target object collected in a high-altitude operation scene are collected and divided into a training set, a validation set, and a test set according to a preset ratio. The preset ratio is set according to actual needs.

[0068] The area where the first target object is located is labeled to obtain a labeled image.

[0069] Step A2: preprocess the annotated image, and train the first image recognition initial model based on the preprocessed annotated image.

[0070] Preprocessing of the labeled image may include denoising, grayscale conversion, and contrast enhancement.

[0071] The model parameters of the first image recognition initial model are set according to the architecture of YOLOv5, and the model parameters can be weight values ​​of the model. The first image recognition initial model can be any one of the YOLOv5s model, the YOLOv5m model, the YOLOv5l model, and the YOLOv5x model.

[0072] Based on the preprocessed labeled images, the first image recognition initial model is trained and iterative training is performed, and each round of iteration includes forward propagation, loss calculation, back propagation and model parameter update.

[0073] Step A3: determine a preset loss function corresponding to the first image recognition initial model, adjust the model parameters of the first image recognition initial model according to the preset loss function, and obtain the first image recognition model.

[0074] Among them, the preset loss function is associated with the one-time target detection algorithm, and the loss terms of the loss function include classification loss, positioning loss and confidence loss.

[0075] Specifically, the preset loss function may be a loss function unique to YOLOv5, which may be a loss function combining classification loss, positioning loss, and confidence loss, and may optimize detection performance. The loss function formula is as follows:

[0076] L=λ cls L cls +λ obj L obj +λ noobj L noobj +λ bbox L bbox .

[0077] Among them, L cls represents the classification loss, L obj and L noobj are the confidence losses for the existence and non-existence of the first target object, L bbox is the bounding box regression loss, λ cls , obj , noobj and λ bbox It is the weight coefficient of each loss item, which is used to balance the importance of each loss item. The category corresponding to the classification loss can be the first target object category and the non-first target object category.

[0078] In multiple rounds of iterations, the model parameters of the first image recognition initial model are adjusted according to the function value of the preset loss function until the function value of the preset loss function converges to obtain the first image recognition model.

[0079] Optionally, the model parameters of the first image recognition initial model are adjusted according to a preset loss function to obtain the first image recognition model:

[0080] Adjust the model parameters of the first image recognition initial model according to the preset loss function to obtain the first image recognition primary intermediate model, train the first image recognition primary intermediate model through the verification image to obtain the first image recognition secondary intermediate model, and train the first image recognition secondary intermediate model through the test image to obtain the first image recognition model.

[0081] The validation image may be an image in a validation set. The model performance of the first image recognition primary intermediate model is evaluated on the validation set. The hyperparameters of the model are adjusted according to the training results. The hyperparameters may include learning rate, batch size, and number of training rounds, etc., to obtain the first image recognition secondary intermediate model.

[0082] The test image can be an image in the test set. The final test is performed on the test set. The first image recognition secondary intermediate model is fine-tuned according to the test results until the preset test requirements are met. The preset test requirements can be set according to actual needs, for example, the loss function converges.

[0083] S250: Determine a target area image according to the recognition result of the first target object.

[0084] S260: Input the target area image into a pre-trained second image recognition model to obtain a recognition result of a second target object associated with the first target object.

[0085] The technical solution of this embodiment is to obtain an image to be identified, perform multi-scale feature extraction on the input image to be identified through the preset deep network and feature pyramid network of the first image recognition model, obtain a feature image, predict the feature image through the head network of the first image recognition model to obtain a preliminary recognition result of the target object, perform non-maximum suppression on the preliminary recognition result to obtain the recognition result of the first target object, determine the target area image according to the recognition result of the first target object, input the target area image into the pre-trained second image recognition model, and obtain the recognition result of the second target object associated with the first target object. The technical solution of the embodiment of the present invention solves the problem of high cost and low efficiency in the current analysis of aerial work through manual inspection and other methods. The target objects in the images collected in the aerial work scene can be automatically identified through a neural network model based on a deep learning algorithm, with high recognition efficiency and low cost, so as to achieve effective analysis of aerial work.

[0086] Figure 3 This is a flowchart of an image recognition method provided in an embodiment of the present invention. This embodiment and the image recognition method in the above embodiment belong to the same inventive concept, and further describes the process of inputting the target area image into a pre-trained second image recognition model to obtain the recognition result of the second target object associated with the first target object. The method can be executed by an image recognition device, which can be implemented by software and / or hardware and integrated into a computer device with application development function.

[0087] like Figure 3 As shown, the image recognition method of this embodiment includes the following steps:

[0088] S310: Acquire an image to be recognized, and input the image to be recognized into a pre-trained first image recognition model to obtain a recognition result of a first target object.

[0089] Among them, the images to be identified include images collected in high-altitude work scenes, and the first image recognition model is a model based on a one-time target detection algorithm or a mask area convolutional neural network algorithm.

[0090] In an optional embodiment, the image to be recognized is input into a pre-trained first image recognition model, and the image to be recognized can be grayscaled to obtain a grayscale processed image to be recognized; the grayscale processed image to be recognized is denoised to obtain a denoised image to be recognized; the denoised image to be recognized is histogram equalized and contrast stretched to improve the contrast of the image to be recognized, highlighting the first target object and the second target object in the image to be recognized, to obtain a contrast enhanced image to be recognized; and the contrast enhanced image to be recognized is input into the pre-trained first image recognition model.

[0091] Enhance the features of the image to be identified through preprocessing and improve the model prediction effect.

[0092] In an optional embodiment, the first image recognition model can be a model based on a masked region convolutional neural network algorithm, and the image to be recognized is input into a pre-trained first image recognition model to obtain a recognition result of the first target object. The image to be recognized can be input into a pre-trained first image recognition model, and the features of the image to be recognized are extracted through a preset feature extraction network of the first image recognition model to obtain a feature image; the feature image is predicted through a region construction network of the first image recognition model to obtain a preliminary recognition result of the first target object; the preliminary recognition result is feature mapped through a region of interest alignment layer to obtain a feature mapping result; and each region of interest of the feature mapping result is predicted through a head network layer to obtain a recognition result of the first target object.

[0093] The Mask R-CNN algorithm is based on the mask region convolutional neural network algorithm. Mask R-CNN is an improved model based on FasterR-CNN. The network structure of the first image recognition Mask R-CNN model includes Backbone network layer, Region Proposal Network (RPN), Region of Interest Alignment Layer (RoIAlign layer) and Head Network Layer (Head layer).

[0094] The Backbone network layer uses ResNet-50 (Residual Network 50, 50 layers of deep residual network) or ResNet-101 (Residual Network 101, 101 layers of deep residual network) as the basic network for feature extraction, and combines the feature pyramid network FPN to extract features at different scales. The RPN layer generates a series of candidate regions (region proposals) that may contain targets based on the feature image and estimates preliminary bounding boxes. RoIAlign layer: replaces RoIPooling in Faster R-CNN, provides feature mapping, and ensures alignment. The Head layer is a two-way parallel Head layer, including a classification head layer, which is used to predict the category of each region of interest RoI, and a bounding box regression head layer, which can refine the predicted bounding box position; it also includes a mask prediction head layer: generates a segmentation mask related to each RoI for instance segmentation.

[0095] The image to be identified is input into the pre-trained first image recognition model, and the preset feature extraction network of the first image recognition model is used. The preset feature extraction network can be a combination of one of ResNet-50 and ResNet-101 and the feature pyramid network FPN. The features of the image to be identified are extracted to obtain a feature image; the feature image is predicted by the region construction network RPN of the first image recognition model to obtain a preliminary recognition result of the first target object, and the preliminary recognition result can be a candidate region of the first target object; the preliminary recognition result is feature mapped by the region of interest alignment layer RoIAlign layer, and features of a fixed size are extracted to obtain a feature mapping result; each region of interest of the feature mapping result is predicted by the head network layer to obtain the recognition result of the first target object. The recognition result of the first target object can be a segmentation mask and a bounding box representing the location of the first target object. Specifically, the segmentation mask of each RoI is output by the mask prediction head layer of the head network layer to realize pixel-level instance segmentation, and non-maximum suppression NMS is applied to reduce redundant detection, and the segmentation mask, category information and bounding box of the first target object are output.

[0096] The training steps of the first image recognition model based on the Mask R-CNN network algorithm are as follows:

[0097] Data preparation: Annotate the bounding box and pixel-level mask of the first target object in the sample image to obtain an annotated image. Input the annotated image into the first image recognition initial model and iteratively train the first image recognition initial model.

[0098] The loss function of the first image recognition initial model can include classification loss, bounding box regression loss and mask segmentation loss. The formula of the loss function is as follows:

[0099] L=L cls +L box +L mask .

[0100] Where L cls is the classification loss, which can be the cross entropy loss, L box is the bounding box regression loss, which can be a smooth L1 loss, L mask is the mask prediction loss, which can be either binary cross entropy or Dice loss.

[0101] Iterate on the training set, each iteration includes forward propagation, loss calculation, back propagation and parameter update, use Adam or SGD optimizer to update the parameters of the model, minimize the loss function, and obtain the first image recognition model.

[0102] After training with sample images, the performance of the first image recognition model can also be evaluated on the validation set, and hyperparameters such as learning rate and positive and negative sample balance can be adjusted based on the evaluation results. The model with the best performance on the validation set is selected, and the model is optimized through the test set to obtain the optimized first image recognition model.

[0103] This embodiment recognizes the images of workers at high altitudes, and realizes dynamic verification of the identity qualifications, protective measures and behavioral norms of workers at high altitudes without interfering with their operations, thereby improving the efficiency of inspection work and the intensity of on-site enforcement, and assisting inspectors in correcting illegal behaviors and quickly collecting evidence at high-altitude work sites.

[0104] S320: Determine a target area image according to the recognition result of the first target object.

[0105] S330. Receive the target area image through the input layer of the second image recognition model.

[0106] Based on the recognition result of the first target object, the area where the first target object is located is cropped and input into the second image recognition model. The second image recognition model can be a CNN model.

[0107] The target area image is received through the input layer of the second image recognition model, and specifically, the preprocessed target area image is received. The preprocessing may include operations such as scaling, normalization, and enhancement to unify the input size and improve the robustness of the model.

[0108] S340. Perform convolution processing on the target area image through the convolution layer of the second image recognition model to obtain a convolution processing result of the target area image.

[0109] The target area image is convolved with a convolution kernel to extract local features in the image. The convolution layer contains multiple convolution kernels to extract different features, such as color, shape, and texture. After the convolution operation, the activation function ReLU is added to introduce nonlinear factors to obtain the convolution processing result.

[0110] S350. Performing pooling processing on the convolution processing result through the pooling layer of the second image recognition model to obtain a pooling processing result.

[0111] The pooling layer is located after the convolution layer, including the maximum pooling layer and the average pooling layer, which is used to perform pooling on the convolution results, reduce the size of the image and the number of parameters, and suppress overfitting.

[0112] S360. Perform feature mapping on the pooling processing result through the fully connected layer of the second image recognition model to obtain a recognition result of the second target object.

[0113] The recognition result of the second target object includes the existence probability of the second target object.

[0114] The fully connected layer is located in the last few layers of the CNN network and is used to map the extracted features to the final classification results. The fully connected layer contains multiple neurons, which are used to output the existence probability of the second target object such as a safety rope or a safety belt.

[0115] The calculation formula of the fully connected layer is as follows:

[0116] Y = f(W*X+b).

[0117] Among them, Y represents the output value, that is, the probability of the existence of a safety rope or a safety belt; f represents the activation function, which can be ReLU; W represents the weight matrix, which is the parameter learned by the model during the training process; X represents the input value, that is, the result of the pooling process; b represents the offset, which is also a parameter learned by the model during the training process.

[0118] The training steps of the second image recognition model may be to collect and annotate a dataset of images containing the second target object collected during aerial work, and divide the annotated dataset into a training set, a validation set, and a test set. Determine the network structure of the second image recognition preliminary model, including the number and parameters of convolutional layers, pooling layers, and fully connected layers, and set training parameters, including learning rate, batch size, and number of iterations. Initialize the weights and offsets of the model; iterate each batch of data in the training dataset, pass the input image through the CNN network, calculate the forward propagation output value, calculate the loss value based on the true label and the model output, calculate the gradient based on the loss value through back propagation, and update the weights and offsets of the model. Evaluate the performance of the model on the validation set, and save the best model as the second image recognition model.

[0119] Optionally, morphological recognition is performed on the second target object based on the morphological features to obtain a morphological recognition result.

[0120] Convert the grayscale image to HSV or HLS color space to identify safety ropes and seat belts of specific colors; set the color threshold to separate the colors of the safety ropes and seat belts from the background; remove noise and refine the outlines of the safety ropes and seat belts through morphological operations such as corrosion, expansion, opening, and closing; analyze the segmented outlines and determine whether they are safety ropes and seat belts based on their shape, length, and width features, and whether they meet the requirements.

[0121] The morphological recognition result can be used together with the recognition result of the second target object output by the model as the basis for safety status judgment.

[0122] The technical solution of this embodiment obtains the image to be identified and inputs the image to be identified into a pre-trained first image recognition model to obtain the recognition result of the first target object, determines the target area image according to the recognition result of the first target object, and receives the target area image through the input layer of the second image recognition model. The target area image is convoluted by the convolution layer of the second image recognition model to obtain the convolution processing result of the target area image, the convolution processing result is pooled by the pooling layer of the second image recognition model to obtain the pooling processing result, and the pooling processing result is feature mapped by the fully connected layer of the second image recognition model to obtain the recognition result of the second target object. The technical solution of the embodiment of the present invention solves the problem of high cost and low efficiency in the current analysis of aerial work through manual inspection and other methods. The target object in the image collected in the aerial work scene can be automatically identified by a neural network model based on the convolutional neural network algorithm, with high recognition efficiency and low cost, so as to realize effective analysis of aerial work.

[0123] Figure 4 This is a schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention. This embodiment can be applied to image recognition scenarios, especially to the recognition of target objects in images collected in aerial work scenarios. The image recognition device can be implemented by software and / or hardware and integrated into a computer terminal device with application development functions.

[0124] like Figure 4 As shown, the image recognition device includes: a first image recognition module 410 , a target area determination module 420 and a second image recognition module 430 .

[0125] Among them, the first image recognition module 410 is used to obtain the image to be recognized, and input the image to be recognized into a pre-trained first image recognition model to obtain the recognition result of the first target object; the target area determination module 420 is used to determine the target area image according to the recognition result of the first target object; the second image recognition module 430 is used to input the target area image into a pre-trained second image recognition model to obtain the recognition result of the second target object associated with the first target object; wherein, the image to be recognized includes an image collected in a high-altitude work scene, and the first image recognition model is a model based on a one-time target detection algorithm or a mask area convolutional neural network algorithm.

[0126] The technical solution of this embodiment obtains the image to be identified and inputs the image to be identified into a pre-trained first image recognition model to obtain the recognition result of the first target object; determines the target area image according to the recognition result of the first target object; inputs the target area image into a pre-trained second image recognition model to obtain the recognition result of the second target object associated with the first target object; wherein the image to be identified includes images collected in aerial work scenes. The technical solution of the embodiment of the present invention solves the problem of high cost and low efficiency in the current analysis of aerial work through manual inspections and other methods. The pre-trained neural network model can be used to automatically identify the target objects in the images collected in the aerial work scenes, with high recognition efficiency and low cost, to achieve effective analysis of aerial work.

[0127] In an optional implementation, when the pre-trained first image recognition model is a model based on a one-shot target detection algorithm, the first image recognition module 410 is specifically used to:

[0128] Through the preset deep network and feature pyramid network of the first image recognition model, multi-scale feature extraction is performed on the input image to be recognized to obtain a feature image; through the head network of the first image recognition model, the feature image is predicted to obtain a preliminary recognition result of the target object; the preliminary recognition result is non-maximum suppressed to obtain a recognition result of the first target object.

[0129] In an optional embodiment, the above device further includes:

[0130] A model training module is used to obtain a target sample image, annotate the area where the first target object is located in the target sample image, and obtain an annotated image; preprocess the annotated image, and train the first image recognition initial model based on the preprocessed annotated image; determine a preset loss function corresponding to the first image recognition initial model, adjust the model parameters of the first image recognition initial model according to the preset loss function, and obtain the first image recognition model; wherein the preset loss function is associated with a one-time target detection algorithm, and the loss terms of the preset loss function include classification loss, positioning loss, and confidence loss.

[0131] In an optional implementation, the second image recognition module 430 is specifically used for:

[0132] The target area image is received through the input layer of the second image recognition model; the target area image is convolved through the convolution layer of the second image recognition model to obtain the convolution result of the target area image; the convolution result is pooled through the pooling layer of the second image recognition model to obtain the pooling result; the pooling result is feature mapped through the fully connected layer of the second image recognition model to obtain the recognition result of the second target object; wherein the recognition result of the second target object includes the existence probability of the second target object.

[0133] In an optional implementation, the first image recognition module 410 is further configured to:

[0134] The image to be identified is gray-scaled to obtain the image to be identified after gray-scale processing; the image to be identified after gray-scale processing is subjected to denoising to obtain the image to be identified after denoising processing; the image to be identified after denoising processing is subjected to histogram equalization and contrast stretching to improve the contrast of the image to be identified, highlight the first target object and the second target object in the image to be identified, and obtain the image to be identified after contrast enhancement; the image to be identified after contrast enhancement is input into the pre-trained first image recognition model.

[0135] The image recognition device provided in the embodiment of the present invention can execute the image recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0136] Figure 5 A schematic diagram of the structure of an image recognition system provided by an embodiment of the present invention.

[0137] like Figure 5 As shown, the image recognition system includes:

[0138] Drone 510 , image transmission module 520 and image recognition module 530 .

[0139] Among them, the drone 510 is equipped with a camera for collecting images to be identified; the image transmission module 520 is used to transmit the images to be identified collected by the drone to the image recognition module; the image recognition module 530 is used to perform image recognition according to the image recognition method provided in any embodiment of the present invention.

[0140] The drone 510 may be a drone with a high-performance flight control system, GPS navigation, and obstacle avoidance sensors (such as laser radar and ultrasonic sensors), and may be equipped with a high-resolution camera to ensure clear images of the high-altitude working area at a specified height.

[0141] The drone flies over the operation area along a preset route, and the camera captures images to be identified at a set frequency.

[0142] The image transmission module 520 integrates high-speed wireless communication technology (such as 5G or private network communication) to transmit the image to be recognized collected by the drone to the image processing and recognition server and terminal management module in real time. After the image to be recognized is compressed, it is transmitted from the image transmission module 520 to the image recognition module 530 via the 5G wireless link. The image recognition module 530 performs pre-processing such as denoising, graying, and contrast enhancement on the image, and then divides the data set to prepare for subsequent analysis.

[0143] The image recognition module 530 is configured with a high-performance GPU server, equipped with a recognition program of the image recognition method provided by any embodiment of the present invention, and can run image preprocessing software and deep learning algorithms, including the YOLOv5 algorithm and the Mask R-CNN algorithm.

[0144] Optionally, the above system further includes:

[0145] The terminal management module is used to provide a preset user interface to display real-time video data, recognition results and security status.

[0146] The terminal management module can store recognition result records and conduct in-depth analysis of real-time video data on operational safety, providing a scientific basis for safety management, helping managers identify safety hazard patterns, optimize safety strategies, and continuously improve operational safety management levels.

[0147] The terminal management module is equipped with an emergency operation interface for handling emergency matters related to the recognition results.

[0148] Optionally, the above system further includes:

[0149] The alarm module is used to send an alarm message to the terminal management module when the recognition result of the second target object is non-existent and the safety belt or safety rope is not recognized.

[0150] Specifically, when the safety belt or safety rope is not recognized, the alarm module sends an alarm message to the terminal management module.

[0151] This embodiment uses the high-definition camera and real-time image transmission technology carried by the drone, and the system can instantly capture the on-site conditions of high-altitude operations. Combined with the automated image recognition algorithm, it significantly shortens the time from image acquisition to safety risk identification, realizes real-time safety monitoring, and greatly improves the emergency response speed. Automated monitoring replaces manual monitoring, reduces the need for on-site monitoring personnel, reduces labor costs, and avoids negligence and errors in manual monitoring, fundamentally reducing the safety risks of operators, ensuring life safety, and reducing labor costs and risks.

[0152] The maneuverability and flexibility of drones enable the system to reach areas that are difficult to reach with traditional monitoring methods, achieving all-round monitoring of high-altitude working environments with comprehensive coverage and strong adaptability.

[0153] The technical solution of this embodiment obtains the image to be identified and inputs the image to be identified into a pre-trained first image recognition model to obtain the recognition result of the first target object; determines the target area image according to the recognition result of the first target object; inputs the target area image into a pre-trained second image recognition model to obtain the recognition result of the second target object associated with the first target object; wherein the image to be identified includes images collected in aerial work scenes. The technical solution of the embodiment of the present invention solves the problem of high cost and low efficiency in the current analysis of aerial work through manual inspections and other methods. The pre-trained neural network model can be used to automatically identify the target objects in the images collected in the aerial work scenes, with high recognition efficiency and low cost, to achieve effective analysis of aerial work.

[0154] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 6 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 6 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. The computer device 12 can be any terminal device with computing capabilities, such as an intelligent controller and server, a mobile phone and other terminal devices.

[0155] like Figure 6 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0156] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0157] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0158] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0159] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0160] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Figure 6 Not shown, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RFID systems, tape drives, and data backup storage systems.

[0161] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing the image recognition method provided by the embodiment of the present invention, which includes:

[0162] Acquire an image to be recognized, and input the image to be recognized into a pre-trained first image recognition model to obtain a recognition result of a first target object;

[0163] Determine a target area image according to a recognition result of the first target object;

[0164] Inputting the target area image into a pre-trained second image recognition model to obtain a recognition result of a second target object associated with the first target object;

[0165] The images to be recognized include images collected in aerial work scenes, and the first image recognition model is a model based on a one-time target detection algorithm or a mask region convolutional neural network algorithm. The embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an image recognition method as provided in any embodiment of the present invention, the method comprising:

[0166] Acquire an image to be recognized, and input the image to be recognized into a pre-trained first image recognition model to obtain a recognition result of a first target object;

[0167] Determine a target area image according to a recognition result of the first target object;

[0168] Inputting the target area image into a pre-trained second image recognition model to obtain a recognition result of a second target object associated with the first target object;

[0169] Among them, the images to be identified include images collected in high-altitude work scenes, and the first image recognition model is a model based on a one-time target detection algorithm or a mask area convolutional neural network algorithm.

[0170] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0171] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0172] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0173] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, Python, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0174] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the image recognition method provided in any embodiment of the present application.

[0175] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, Python, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0176] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0177] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An image recognition method, characterized in that: include: Acquire an image to be recognized, and input the image to be recognized into a pre-trained first image recognition model to obtain a recognition result of a first target object; Determine a target area image according to the recognition result of the first target object; Inputting the target area image into a pre-trained second image recognition model to obtain a recognition result of a second target object associated with the first target object; Among them, the images to be identified include images collected in high-altitude work scenarios, and the first image recognition model is a model based on a one-time target detection algorithm or a mask area convolutional neural network algorithm.

2. The method according to claim 1, characterized in that: In the case where the pre-trained first image recognition model is a model based on a one-time target detection algorithm, inputting the image to be recognized into the pre-trained first image recognition model to obtain a recognition result of the first target object includes: Performing multi-scale feature extraction on the input image to be recognized through the preset deep network and feature pyramid network of the first image recognition model to obtain a feature image; Predicting the feature image through the head network of the first image recognition model to obtain a preliminary recognition result of the target object; Non-maximum suppression is performed on the preliminary recognition result to obtain a recognition result of the first target object.

3. The method according to claim 2, characterized in that The process of training the first image recognition model includes: Acquire a target sample image, and mark the area where the first target object is located in the target sample image to obtain a marked image; Preprocessing the annotated image, and training a first image recognition initial model based on the preprocessed annotated image; Determine a preset loss function corresponding to the first image recognition initial model, and adjust model parameters of the first image recognition initial model according to the preset loss function to obtain the first image recognition model; The preset loss function is associated with the one-time target detection algorithm, and the loss terms of the preset loss function include classification loss, positioning loss and confidence loss.

4. The method according to claim 1, characterized in that The step of inputting the target area image into a pre-trained second image recognition model to obtain a recognition result of a second target object associated with the first target object includes: Receiving the target area image through an input layer of a second image recognition model; Performing convolution processing on the target area image through the convolution layer of the second image recognition model to obtain a convolution processing result of the target area image; Performing pooling processing on the convolution processing result through the pooling layer of the second image recognition model to obtain a pooling processing result; Performing feature mapping on the pooling processing result through the fully connected layer of the second image recognition model to obtain a recognition result of the second target object; The recognition result of the second target object includes the existence probability of the second target object.

5. The method according to claim 1, characterized in that The step of inputting the image to be recognized into a pre-trained first image recognition model comprises: Performing grayscale processing on the image to be identified to obtain the image to be identified after grayscale processing; Performing noise reduction processing on the image to be identified after the grayscale processing to obtain the image to be identified after the noise reduction processing; Performing histogram equalization and contrast stretching on the image to be identified after the noise reduction process to improve the contrast of the image to be identified, highlighting the first target object and the second target object in the image to be identified, and obtaining the image to be identified after contrast enhancement; The image to be recognized after contrast enhancement is input into a pre-trained first image recognition model.

6. An image recognition device, characterized in that: include: A first image recognition module is used to obtain an image to be recognized, and input the image to be recognized into a pre-trained first image recognition model to obtain a recognition result of a first target object; A target area determination module, used to determine a target area image according to the recognition result of the first target object; A second image recognition module, inputting the target area image into a pre-trained second image recognition model to obtain a recognition result of a second target object associated with the first target object; Among them, the images to be identified include images collected in high-altitude work scenarios, and the first image recognition model is a model based on a one-time target detection algorithm or a mask area convolutional neural network algorithm.

7. An image recognition system, characterized in that: include: UAV, image transmission module and image recognition module; Wherein, the drone is equipped with a camera for collecting images to be identified; The image transmission module is used to transmit the image to be recognized collected by the drone to the image recognition module; The image recognition module is used to perform image recognition according to the image recognition method according to any one of claims 1-5.

8. A computer device, characterized in that: The computer device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image recognition method as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image recognition method as described in any one of claims 1 to 5 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the image recognition method according to any one of claims 1 to 5.

Images