Method and system for monitoring high-altitude operation status based on sensor data analysis

Abstract

The present invention provides a method and system for monitoring the state of aerial work based on sensor data analysis. The model is adjusted by joint learning, not only the second operation sensor monitoring image and the corresponding prior monitoring target state for training the aerial work state monitoring neural network are obtained, but also the prior involved image recognition result of the second operation sensor monitoring image is obtained; then, after the implicit representation mining of the second operation sensor monitoring image, the third monitoring implicit representation obtained is loaded into the second branch network component for state detection of the operation sensor monitoring image, and loaded into the involved image recognition neural network for involved semantic analysis. Then, the adjustment cost value obtained by combining the state detection of the operation sensor monitoring image and the involved semantic analysis is used to adjust the network parameters of the aerial work state monitoring neural network. In this way, the ability of the aerial work state monitoring neural network to understand the aerial work image is increased, and the accuracy of the operation state detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and system for monitoring the status of aerial work based on sensor data analysis. Background Art

[0002] In today's various aerial work scenarios, such as construction, power maintenance, bridge construction and other fields, it is crucial to ensure the safety of workers. The aerial work environment is complex and changeable, and there are many potential risks. For example, workers fail to wear safety equipment correctly, safety equipment fails, and there are safety hazards in the working environment, which may cause serious safety accidents. Therefore, real-time and accurate monitoring of the aerial work status, timely detection of potential safety problems and taking corresponding measures have become the key links to ensure the safety of aerial work. Existing aerial work status monitoring methods face many challenges in practical applications. On the one hand, traditional manual monitoring methods are not only inefficient, but also limited by manpower and monitoring range, making it difficult to achieve comprehensive and real-time monitoring. At the same time, manual monitoring is easily affected by subjective factors, which makes it difficult to ensure the accuracy and reliability of the monitoring results. On the other hand, with the development of technology, some monitoring methods based on sensors and image processing have gradually been applied. However, these methods have the problem of insufficient image understanding ability when dealing with complex aerial work scenes. Summary of the invention

[0003] In view of this, the present invention provides a method and system for monitoring the status of aerial work based on sensor data analysis. The technical solution of the present invention is implemented as follows:

[0004] On the one hand, the present invention provides a method for monitoring the state of aerial work based on sensor data analysis, which is used to perform state detection on the operation sensor monitoring image using a high-altitude operation state monitoring neural network, wherein the high-altitude operation state monitoring neural network includes a first branch network component and a second branch network component. The state detection method includes: acquiring a first operation sensor monitoring image; loading the first operation sensor monitoring image into the first branch network component for data feature representation, and obtaining a first monitoring implicit representation corresponding to the first operation sensor monitoring image; loading the first monitoring implicit representation into the second branch network component, and performing monitoring target state recognition on the first operation sensor monitoring image to obtain a monitoring target state recognition result, wherein the monitoring target state recognition result is used to indicate the state relationship of the monitoring target in the first operation sensor monitoring image.

[0005] Wherein, the high-altitude work status monitoring neural network is calibrated through the following steps: obtaining a second work sensor monitoring image, a priori monitoring target state corresponding to the monitoring target in the second work sensor monitoring image, and a priori involved image recognition result of the second work sensor monitoring image; wherein, the prior monitoring target state is used to indicate the state relationship of the monitoring target in the second work sensor monitoring image, and the prior involved image recognition result is used to indicate the image recognition result of the involved image recognition of the second work sensor monitoring image; loading the second work sensor monitoring image into the first branch network component for data feature representation, and obtaining a third monitoring implicit representation corresponding to the second work sensor monitoring image; the third monitoring implicit representation Load it into the second branch network component, perform monitoring target state recognition on the second operation sensor monitoring image, and obtain the monitoring target state recognition result corresponding to the monitoring target in the second operation sensor monitoring image; load the third monitoring implicit representation into the involved image recognition neural network, perform involved image recognition processing on the second operation sensor monitoring image, and obtain the involved image recognition result corresponding to the second operation sensor monitoring image; based on the monitoring target state recognition result and the prior monitoring target state, as well as the involved image recognition result and the prior involved image recognition result, obtain the calibrated adjustment cost value; based on the adjustment cost value, adjust the network parameters of the first branch network component and the second branch network component.

[0006] On the other hand, the present invention provides a computer system, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps in the above method when executing the program.

[0007] Beneficial effects of the present invention: The present invention uses joint learning to perform model training to calibrate the high-altitude work status monitoring neural network, not only obtaining the second work sensor monitoring image and the corresponding prior monitoring target state for training the high-altitude work status monitoring neural network, but also obtaining the prior involved image recognition result of the second work sensor monitoring image; then after the second work sensor monitoring image is mined for implicit representation, the third monitoring implicit representation obtained is loaded into the second branch network component for state detection of the work sensor monitoring image, and loaded into the involved image recognition neural network for involved semantic analysis. Then, the calibration cost values ​​obtained by combining the state detection of the work sensor monitoring image and the involved semantic analysis are used to adjust the network parameters of the high-altitude work status monitoring neural network. In this way, the high-altitude work status monitoring neural network's ability to understand the high-altitude work image is increased, and the accuracy of work status detection is improved.

[0008] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present invention and, together with the specification, are used to explain the technical solutions of the present invention.

[0010] Figure 1 A schematic diagram of the implementation flow of a method for monitoring the status of aerial work based on sensor data analysis provided in an embodiment of the present invention.

[0011] Figure 2 A hardware entity schematic diagram of a computer system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0013] The embodiment of the present invention provides a method for monitoring the state of aerial work based on sensor data analysis, which can be executed by a processor of a computer system, wherein the computer system can refer to a device with data processing capabilities such as a server, a laptop, a tablet computer, and a desktop computer.

[0014] The embodiment of the present invention provides a method for monitoring the state of aerial work based on sensor data analysis, which is used to detect the state of the operation sensor monitoring image using a neural network for monitoring the state of aerial work. The neural network for monitoring the state of aerial work includes a first branch network component and a second branch network component, such as Figure 1 As shown, the method includes:

[0015] Step S100: Acquire a first operation sensor monitoring image.

[0016] The high-altitude work status monitoring neural network is mainly composed of the first branch network component and the second branch network component. The first branch network component is used to represent the data features of the input work sensor monitoring image, and can mine the various feature information contained in the image, and convert the image data into an implicit representation that is easier for the computer system to process and analyze. For example, for an image containing elements such as workers operating tools and safety ropes in a high-altitude work scene, the first branch network component can extract the shape, color, position and other features of these elements and convert them into corresponding implicit representations. The second branch network component identifies the monitoring target state of the image based on the implicit representation obtained by the first branch network component, and determines the state relationship of the monitoring target (objects related to work safety, such as safety helmets, seat belts, etc.) in the image.

[0017] When the computer system executes step S100, a first operation sensor monitoring image is obtained. This process can be achieved through a variety of technical means. First, multiple image acquisition devices, such as high-definition cameras, are arranged at the aerial work site to collect images at certain time intervals or when triggered by specific events. These cameras are reasonably installed in locations that can fully capture the work scene, such as around the work platform, on appropriate floors of buildings, etc., to ensure that the collected images can cover all areas and objects related to the safety of aerial work.

[0018] The collected image data will be transmitted to the computer system through a wired network (such as Ethernet) or a wireless network (such as Wi-Fi, 4G / 5G). After receiving the image data, the computer system performs preliminary preprocessing operations on it. For example, grayscale processing is performed on the image to convert the color image into a grayscale image to simplify subsequent calculations and analysis. The grayscale formula is: Gray = 0.299×R+0.587×G+0.114×B, where R, G, and B represent the values ​​of the red, green, and blue channels in the color image, respectively, and Gray is the converted grayscale value.

[0019] Next, the computer system performs noise reduction on the image to remove the noise interference introduced during the image acquisition and transmission process and improve the image quality. Common noise reduction algorithms such as Gaussian filtering smooth the image and reduce the impact of noise by performing weighted averaging on each pixel in the image and its neighboring pixels.

[0020] In order to enable the image data to be effectively processed by the aerial work status monitoring neural network, the computer system will also normalize the image. The purpose of normalization is to map the pixel value of the image to a specific range, such as [0,1] or [-1, 1], which can speed up the training convergence of the neural network and improve the stability and generalization ability of the model. The normalization formula is: Normalized = (Pixel-Min) / (Max-Min), where Pixel is the original pixel value, Min and Max are the minimum and maximum pixel values ​​in the image, respectively, and Normalized is the normalized pixel value.

[0021] In actual aerial work scenarios, the first operation sensor monitoring image may contain many different types of information. For example, the image may contain workers performing aerial work, wearing safety equipment such as helmets and safety belts, and surrounded by objects such as work tools and lifting platforms. The state of these objects and the relationship between them are crucial to determining whether aerial work is safe. The first operation sensor monitoring image acquired by the computer system is to contain this key information so that it can be analyzed and judged later through the aerial work state monitoring neural network.

[0022] The acquisition of the first operation sensor monitoring image by the computer system is the starting step of the entire high-altitude operation status monitoring process. Through reasonable image acquisition, transmission, preprocessing and other technical means, it provides a high-quality data foundation for subsequent image feature extraction, status recognition and other operations, ensuring that the high-altitude operation status can be accurately monitored, ensuring the safety of operators and the smooth progress of operations.

[0023] Step S200: Load the first operation sensor monitoring image into the first branch network component for data feature representation, and obtain a first monitoring implicit representation corresponding to the first operation sensor monitoring image.

[0024] When the computer system executes step S200, the first operation sensor monitoring image is loaded into the first branch network component for data feature representation, thereby obtaining the first monitoring implicit representation corresponding to the first operation sensor monitoring image. This process involves complex and sophisticated image processing and analysis technology, which aims to extract feature information from the image that is valuable for subsequent monitoring target state recognition and convert it into an implicit representation suitable for neural network processing.

[0025] When performing this step, the computer system uses Convolutional Neural Network (CNN) technology. CNN is a deep learning model architecture specifically designed to process image data. It automatically learns features in images through components such as convolutional layers, pooling layers, and fully connected layers. In the first branch network component, the convolution layer performs a convolution operation on the first operation sensor monitoring image. The core idea of ​​the convolution operation is to use a small convolution kernel (usually a square matrix) to slide on the image and perform weighted summation of the pixel values ​​at each position to generate a new feature map. For example, for a convolution kernel of size 3x3 and an image pixel matrix, the central element of the convolution kernel is multiplied by the pixel value at the corresponding position on the image, and the surrounding elements are also multiplied by the pixel value at the corresponding position, and then all the products are added to obtain a new value for the corresponding position on the feature map. The calculation formula can be expressed as:

[0026] ;

[0027] in, is the pixel value at position (x, y) on the feature map, K(i, j) is the element value of the convolution kernel at position (i, j), I(x+i, y+j) is the pixel value of the image at position (x+i, y+j), and m and n are the number of rows and columns of the convolution kernel, respectively.

[0028] Through different convolution kernels, various features in the image can be extracted, such as edges, textures, shapes, etc. For example, a horizontal convolution kernel can highlight the horizontal edges in the image, while a vertical convolution kernel can capture the vertical edges. Multiple different convolution kernels working in parallel can extract rich and diverse feature information from the image.

[0029] After extracting features through the convolution layer, the computer system uses the pooling layer to downsample the feature map. The purpose of the pooling operation is to reduce the amount of data and reduce computational complexity while retaining the main features of the image. Common pooling methods include max pooling and average pooling. Taking max pooling as an example, it selects the maximum value in a window of fixed size as the output after pooling. For example, for a 2x2 max pooling window, it selects the largest value among the 4 pixel values ​​in this window as the output, thereby reducing the size of the feature map by half. The calculation formula for max pooling can be simply expressed as:

[0030] ;

[0031] Among them, P(x,y) is the pixel value of the feature map at position (x,y) after pooling, F(x×s+i,y×s+j) is the pixel value of the feature map at the corresponding position before pooling, and s is the size of the pooling window.

[0032] Average pooling calculates the average value of all pixel values ​​in the window as the output. The pooling operation not only reduces the amount of data, but also has a certain degree of translation invariance, making the model more robust to changes in the position of objects in the image.

[0033] After convolution and pooling operations, the computer system passes the obtained feature map to the fully connected layer. The function of the fully connected layer is to synthesize and integrate all the features extracted previously to form a complete feature representation. Each neuron in the fully connected layer is connected to all neurons in the previous layer, and the output is calculated through the weight matrix and bias term. The calculation formula is:

[0034] ;

[0035] Among them, y is the output of the fully connected layer, w i is the weight, x i is the input feature, b is the bias term, is an activation function, such as the ReLU (Rectified Linear Unit) function, which is defined as (x) = max(0, x). The role of the activation function is to introduce nonlinear factors into the neural network, so that the model can learn more complex functional relationships.

[0036] Through the above series of operations, the computer system finally converts the first operation sensor monitoring image into the first monitoring implicit representation. This implicit representation is a high-dimensional vector that contains a comprehensive expression of various feature information in the image. For example, for an image containing a high-altitude operation scene, after being processed by the first branch network component, the first monitoring implicit representation may contain the posture characteristics of the worker, the presence and status characteristics of the safety equipment, the layout characteristics of the working environment, etc. These feature information is encoded in the implicit representation, providing a key data basis for the subsequent second branch network component to identify the monitoring target state.

[0037] In practical applications, the structure and parameters of the first branch network component are usually obtained through training with a large amount of sample data. The computer system uses the backpropagation algorithm to adjust the weights and bias items in the network so that the network can better extract features from the image. The backpropagation algorithm calculates the gradient of the loss function to the network parameters, and then updates the parameters according to the gradient direction to minimize the value of the loss function. During the training process, the computer system continuously inputs sample images into the first branch network component, and adjusts the network parameters according to the difference between the output result and the true label until the network can accurately extract the features of the image and generate an effective first monitoring implicit representation.

[0038] The computer system executes step S200, and converts the first operation sensor monitoring image into a first monitoring implicit representation containing rich feature information through operations such as convolution, pooling and fully connected layers in the convolutional neural network technology, providing solid data support for subsequent monitoring target state identification, ensuring that the high-altitude operation state monitoring neural network can accurately analyze and judge the state relationship of the monitoring targets in the image, thereby realizing effective monitoring of the high-altitude operation state.

[0039] Step S300: Load the first monitoring implicit representation into the second branch network component, perform monitoring target state identification on the first operation sensor monitoring image, and obtain a monitoring target state identification result, wherein the monitoring target state identification result is used to indicate the state relationship of the monitoring target in the first operation sensor monitoring image.

[0040] When the computer system executes step S300, based on the first monitoring implicit representation, the first operation sensor monitoring image is subjected to monitoring target state recognition, thereby obtaining a monitoring target state recognition result. This step is a key link in the aerial operation state monitoring process, and is intended to accurately determine the state relationship of the monitoring target related to the operation safety in the image based on the first monitoring implicit representation output by the first branch network component.

[0041] The computer system first utilizes the powerful feature analysis and classification capabilities of the second branch network component. The second branch network component usually contains multiple neural network layers that are carefully designed and trained to perform in-depth analysis of the input first monitoring implicit representation. In actual operation, the computer system passes the first monitoring implicit representation as input to the second branch network component.

[0042] The neurons in the second branch network component perform weighted calculations on each feature dimension in the first monitoring implicit representation. Each neuron has a corresponding weight value, which is continuously adjusted and optimized during the network training process so that the network can properly weigh different features. For example, for a first monitoring implicit representation that contains information such as the posture of a worker working at heights and the wearing of safety equipment, the weight value can make the features related to the wearing of safety equipment have a higher influence in judging the state of the monitoring target.

[0043] Next, the result of the weighted calculation is processed by the activation function. The role of the activation function is to introduce nonlinear factors into the neural network, so that the network can learn more complex functional relationships, thereby improving its ability to distinguish different states. Common activation functions include the ReLU (Rectified Linear Unit) function, whose expression is f(z) = max(0,z). Through the activation function, neurons can decide whether to activate based on the value of the weighted input, thereby passing information to the next layer. For example, when the weighted input z is a negative number, the output after processing by the ReLU function is 0, indicating that the neuron is not activated; when z is a positive number, the output is z itself, the neuron is activated and transmits information.

[0044] As data is gradually transmitted through the layers of the second branch network component, the network will continuously extract and integrate higher-level feature information. These high-level feature information can more comprehensively reflect the status of the monitoring target. For example, when judging whether a worker is wearing a safety belt correctly during high-altitude operations, the features extracted after multiple layers of processing may no longer be just low-level features such as the color and shape of the safety belt, but a combination of high-level feature information such as the relative position of the safety belt to the worker's body and the connection status.

[0045] After a series of feature processing and analysis, the output layer of the second branch network component will give the monitoring target state recognition result. The output layer usually uses the Softmax function to convert the network output into a probability distribution to represent the possibility of different monitoring target states. The formula of the Softmax function is ,in represents the probability that the monitoring target is in state j under the input x (i.e., the first monitoring implicit representation), z j is the unnormalized score of the output layer corresponding to state j, and K is the number of all possible monitoring target states. Through the Softmax function, the computer system can obtain the probability value of each monitoring target state, and the state with the highest probability is the final monitoring target state recognition result.

[0046] For example, in an image of a high-altitude operation, the monitoring targets include whether the worker is wearing a helmet, whether the safety belt is fastened, etc. After the computer system inputs the first monitoring implicit representation into the second branch network component, after calculation and processing at each layer, the output layer obtains the probability of the helmet wearing state being "worn" as 0.9 and the probability of "not worn" as 0.1 through the Softmax function; the probability of the safety belt state being "fastened" is 0.8 and the probability of "not fastened" is 0.2. Then, the computer system determines the monitoring target state recognition result based on the probability value that the worker is wearing a helmet and the safety belt is fastened.

[0047] In order to achieve accurate recognition of the state of the monitoring target, the second branch network component needs to undergo a lot of training. During the training process, the computer system uses a large number of annotated second operation sensor monitoring images, which contain the real state information of the monitoring target. By comparing the recognition results obtained after these images are processed by the first branch network component and the second branch network component with the real state, a loss function (such as the cross entropy loss function) is used to compare the recognition results obtained by the first branch network component and the second branch network component with the real state. , where N is the number of samples, is the label (0 or 1) of the true state of sample i j, The error is calculated by using the probability that sample i is predicted to be state j). The computer system then uses the back-propagation algorithm to adjust the weight values ​​in the second branch network component according to the gradient of the loss function, so that the network can more accurately identify the monitoring target state in subsequent processing.

[0048] In step S300, the computer system performs in-depth feature analysis, calculation and processing on the first monitoring implicit representation through the second branch network component, uses technical means such as activation function and Softmax function, and combines the training optimization process to finally obtain a monitoring target state recognition result that accurately reflects the monitoring target state relationship in the first operation sensor monitoring image, providing strong data support and decision-making basis for ensuring the safety of high-altitude operations.

[0049] In the embodiment of the present invention, the aerial work status monitoring neural network can be adjusted through the following steps S10 to S60:

[0050] Step S10: Acquire a second work sensor monitoring image, a priori monitoring target state corresponding to the monitoring target in the second work sensor monitoring image, and a priori involved image recognition result of the second work sensor monitoring image; wherein the prior monitoring target state is used to indicate the state relationship of the monitoring target in the second work sensor monitoring image, and the prior involved image recognition result is used to indicate the image recognition result of the involved image recognition of the second work sensor monitoring image.

[0051] When the computer system executes step S10, it obtains the second operation sensor monitoring image, the prior monitoring target state corresponding to the monitoring target in the image, and the prior involved image recognition result of the second operation sensor monitoring image. This step is an important starting link in the neural network calibration process of high-altitude operation status monitoring, and provides the necessary data basis for subsequent network training.

[0052] Computer systems use a variety of methods to obtain sensor monitoring images of the second operation. Usually, multiple image acquisition devices, such as high-definition cameras, are reasonably arranged at the aerial work site. The installation locations of these cameras are carefully planned, such as being installed on fixed brackets around the work area, or attached to specific locations on the work platform to ensure that the aerial work scene can be fully and clearly captured. The camera captures images at predetermined time intervals, such as every few seconds, or when triggered by specific events, such as when a worker starts a critical operation. The collected image data is transmitted to the computer system in real time via a wired network (such as Ethernet) or a wireless network (such as Wi-Fi, 4G / 5G).

[0053] Take a high-altitude operation scene in a construction site as an example. There are multiple tower cranes lifting construction materials on site, and workers are doing wall construction and installation operations on scaffolding at different heights. Cameras installed near the tower crane brackets, scaffolding, and around the building can capture workers' operating movements, tools used, safety equipment worn, and surrounding environment information. These images will be transmitted to the computer system as the second operation sensor monitoring image.

[0054] To obtain the prior monitoring target state corresponding to the monitoring target in the second operation sensor monitoring image, professionals are required to annotate the collected images in detail. The monitoring targets are objects related to work safety, such as safety helmets, safety belts, and the correct placement of tools. The prior monitoring target state clarifies the state relationship of these monitoring targets in the image. For example, for an image showing a worker working on a scaffold, a professional annotator will mark whether the worker is wearing a safety helmet correctly (worn or not), whether the safety belt is fastened (fastened or not), whether the tool in his hand is in a safe operating position (yes or no), and other information. These annotation information is the prior monitoring target state. The annotation process can be completed using a special image annotation tool by directly marking on the image, adding text annotations, etc.

[0055] For the prior involved image recognition results, the computer system determines them with the help of specific image recognition technology and expertise. Involved image recognition aims to identify other elements or scene features in the image that are associated with the monitoring target. For example, in the above-mentioned construction images, in addition to paying attention to workers and safety equipment, it is also necessary to identify relevant elements in the surrounding environment, such as whether the scaffolding is erected in accordance with the specifications, whether there are dangerous areas in the lifting path of the tower crane, etc. The computer system can use pre-trained image recognition models that have been trained on a large amount of relevant image data and can recognize common objects and features in various scenes. By inputting the second operation sensor monitoring image into these models, combined with some post-processing algorithms, such as target detection algorithms (such as the YOLO algorithm based on deep learning: ,in is the probability of belonging to a certain type of target at position (x, y), is the weight, is the feature, b is the bias term, is an activation function) to determine the categories and locations of various objects in the image, and then obtain the prior image recognition results. For example, it can be recognized that the scaffolding in the image belongs to the standard construction type, and there are no obstacles on the tower crane lifting path.

[0056] Step S20: Load the second operation sensor monitoring image into the first branch network component for data feature representation, and obtain a third monitoring implicit representation corresponding to the second operation sensor monitoring image.

[0057] When the computer system executes step S20, the second operation sensor monitoring image is loaded into the first branch network component for data feature representation, thereby obtaining the third monitoring implicit representation corresponding to the second operation sensor monitoring image. This step is a key link in the neural network calibration process for aerial work status monitoring, which aims to extract deep and valuable feature information from the second operation sensor monitoring image and convert it into an implicit representation suitable for subsequent processing of the neural network.

[0058] The computer system first inputs the second operation sensor monitoring image into the first branch network component. The first branch network component usually adopts a convolutional neural network (CNN) architecture, the core of which is the convolution layer. The convolution layer slides on the image through the convolution kernel to extract features from each local area of ​​the image. Different types of features in the image can be extracted through different convolution kernels. For example, the horizontal convolution kernel can highlight the horizontal edge information in the image, while the vertical convolution kernel helps to capture the vertical edges. For a second operation sensor monitoring image containing a high-altitude operation scene, after the convolution operation, the edge features of the worker's body contour, the shape features of the safety equipment, etc. may be extracted.

[0059] After the convolution layer, computer systems usually apply a pooling layer. The purpose of the pooling operation is to downsample the feature map and reduce the amount of data while retaining the main features and improving the robustness of the model. After the convolution and pooling operations, the resulting feature map is passed to the fully connected layer. The fully connected layer integrates all the features to form a complete feature representation.

[0060] Through the above series of operations, the computer system finally converts the second operation sensor monitoring image into the third monitoring implicit representation. This implicit representation is a high-dimensional vector that integrates various feature information in the image. For example, for an aerial work image, the third monitoring implicit representation may include the worker's action posture characteristics, the color and shape characteristics of the safety equipment, the layout characteristics of the working environment, etc. These feature information is encoded in the implicit representation, which provides core data support for the subsequent second branch network component to identify the monitoring target state and the semantic analysis involving the image recognition neural network.

[0061] In practical applications, the parameters of the first branch network component (such as the weight of the convolution kernel, the weight matrix of the fully connected layer, and the bias term) are obtained through training with a large amount of sample data. The computer system uses the back-propagation algorithm to calculate the gradient of the loss function to these parameters, and adjusts the parameters according to the gradient to minimize the value of the loss function. During the training process, the computer system inputs a large number of second operation sensor monitoring images into the first branch network component, and by continuously adjusting the parameters, the network can accurately extract the features of the image and generate an effective third monitoring implicit representation. For example, when faced with high-altitude work images at different angles and under different lighting conditions, the fully trained first branch network components can extract stable and representative features and generate reliable third monitoring implicit representations, providing a solid foundation for subsequent high-altitude work status monitoring and related analysis.

[0062] Step S30: Load the third monitoring implicit representation into the second branch network component, perform monitoring target state recognition on the second operation sensor monitoring image, and obtain a monitoring target state recognition result corresponding to the monitoring target in the second operation sensor monitoring image.

[0063] When the computer system executes step S30, the third monitoring implicit representation is loaded into the second branch network component, and the monitoring target state recognition is performed on the second operation sensor monitoring image to obtain the monitoring target state recognition result corresponding to the monitoring target in the second operation sensor monitoring image. This step is based on the third monitoring implicit representation extracted previously, and uses the powerful analysis and recognition capabilities of the second branch network component to accurately determine the state relationship of the monitoring target in the image.

[0064] The second branch network component is a well-designed and trained deep neural network that can perform in-depth feature analysis and pattern matching on the input third monitoring implicit representation. After the computer system passes the third monitoring implicit representation as input to the second branch network component, the component first further extracts and transforms its features.

[0065] Inside the second branch network component, there may be multiple hidden layers. The neurons in these hidden layers perform complex nonlinear transformations on the third monitoring implicit representation of the input. Each neuron is connected to each dimension of the input through a weight, and the weight determines the degree of influence of each input dimension on the neuron output.

[0066] As data is gradually transmitted in the hidden layer of the second branch network component, the network will continuously extract and integrate higher-level feature information. These high-level feature information can more comprehensively reflect the status of the monitoring target. For example, in a high-altitude work scenario, the monitoring targets may include whether the worker wears a helmet correctly, whether the seat belt is fastened, etc. After processing in the hidden layer, the network can extract from the third monitoring implicit representation not only the appearance features of the helmet and the seat belt, but also their relative position to the worker's body, connection status and other high-level feature information. These feature information is crucial for accurately judging the status of the monitoring target.

[0067] After feature processing in multiple hidden layers, the data reaches the output layer of the second branch network component. The output layer usually uses the Softmax function to convert the output of the network into a probability distribution to represent the possibility of different monitoring target states. Through the Softmax function, the computer system can obtain the probability value of each monitoring target state. The state with the highest probability is the final monitoring target state recognition result. For example, for a second operation sensor monitoring image, the monitoring target is the state of the worker wearing a safety helmet. After processing by the second branch network component, the output layer obtains the probability of the worker "wearing a safety helmet" as 0.9 and the probability of "not wearing a safety helmet" as 0.1 through the Softmax function. Then, the computer system judges the monitoring target state recognition result as the worker wearing a safety helmet based on the probability value.

[0068] In order to ensure that the second branch network component can accurately identify the state of the monitoring target, it needs to be trained on a large amount of sample data. During the training process, the computer system uses a large number of annotated second operation sensor monitoring images, which contain the real state information of the monitoring target. The recognition results obtained after these images are processed by the first branch network component and the second branch network component are compared with the real state, and the error is calculated using a loss function (such as a cross entropy loss function).

[0069] The computer system then uses the back-propagation algorithm to adjust the weight values ​​in the second branch network component according to the gradient of the loss function. The back-propagation algorithm calculates the gradient of the loss function to the weights of each layer of the network, and then adjusts the weights in the opposite direction of the gradient, so that the value of the loss function gradually decreases. During the continuous training process, the second branch network component gradually learns how to accurately extract features from the third monitoring implicit representation to identify the state of the monitoring target.

[0070] Step S40: Load the third monitoring implicit representation into the involved image recognition neural network, perform involved image recognition processing on the second operation sensor monitoring image, and obtain the involved image recognition result corresponding to the second operation sensor monitoring image.

[0071] When the computer system executes step S40, the third monitoring implicit representation is loaded into the involved image recognition neural network, and the involved image recognition processing is performed on the second operation sensor monitoring image, thereby obtaining the involved image recognition result corresponding to the second operation sensor monitoring image. This step is intended to understand the second operation sensor monitoring image from a broader semantic level, mine other important information associated with the monitoring target in the image, and further enrich the cognition of the aerial work scene.

[0072] The image recognition neural network is a neural network model designed specifically for this type of task. It can perform targeted analysis on the third-party implicit representation of the input to identify various semantic relationships and related features in the image. After the computer system passes the third-party implicit representation as input to the image recognition neural network, the network processes the input according to its own structure and algorithm.

[0073] Assume that the involved image recognition neural network includes multiple convolutional layers and fully connected layers. In the convolutional layer part, similar to the convolution operation in the first branch network component, different convolution kernels are used to perform convolution operations on the third monitoring implicit representation. Since the third monitoring implicit representation is already a feature representation processed by the first branch network component, the convolution operation here is intended to further extract features related to the involved image recognition. Different types of features in the third monitoring implicit representation can be captured by different convolution kernels. For example, some convolution kernels may be sensitive to environmental features around the monitoring target, while others may pay more attention to object features that are specifically associated with the monitoring target. After feature extraction by the convolutional layer, the computer system uses a pooling layer to downsample the generated feature map. The purpose of the pooling operation is to reduce the amount of data while retaining the main features and improving the robustness of the model. As the data flows in the involved image recognition neural network, the feature map after convolution and pooling processing will be passed to the fully connected layer. The fully connected layer synthesizes and integrates all features to form a complete feature representation for subsequent classification or reasoning.

[0074] In practical applications, the training of the involved image recognition neural network is based on a large number of annotated second-operation sensor monitoring images and their corresponding prior involved image recognition results. The computer system uses the back-propagation algorithm to calculate the gradient of the loss function to the network parameters (such as convolution kernel weights, fully connected layer weight matrices, and bias terms), and adjusts these parameters according to the gradient to minimize the value of the loss function. Through continuous training, the involved image recognition neural network can learn how to accurately extract features related to the involved image from the third monitoring implicit representation, and accurately identify and classify them. For example, in a high-altitude operation scene, for a second-operation sensor monitoring image, the involved image recognition neural network can identify information such as the building structure around the operation area, the layout of other construction equipment, etc. in addition to the monitoring targets (such as workers and safety equipment) in the image. This information is of great significance for a comprehensive assessment of the safety and environmental conditions of high-altitude operations.

[0075] Step S50: obtaining a calibration cost value based on the monitoring target state recognition result and the prior monitoring target state, and the involved image recognition result and the prior involved image recognition result.

[0076] When the computer system executes step S50, the adjustment cost value of the adjustment is obtained based on the monitoring target state recognition result and the prior monitoring target state, as well as the involved image recognition result and the prior involved image recognition result. This step plays a key role in the adjustment process of the high-altitude operation state monitoring neural network. It quantifies the difference between the current performance of the network and the expected performance through comparative analysis of different recognition results, and provides a basis for the subsequent adjustment of network parameters.

[0077] For the monitoring target state recognition result and the prior monitoring target state, the computer system will first evaluate the degree of difference between the two. The prior monitoring target state is known information that accurately reflects the actual state of the monitoring target in the second operation sensor monitoring image, while the monitoring target state recognition result is the judgment made by the aerial work state monitoring neural network after analyzing the same image. The computer system uses appropriate metrics to measure the difference between the two. For example, for a simple two-classification problem, the monitoring target is whether the worker wears a safety helmet. The prior monitoring target state shows that the worker wears a safety helmet (marked as 1), while the monitoring target state recognition result predicts that the worker does not wear a safety helmet (marked as 0). In this case, there is a clear difference between the two.

[0078] Next, the computer system performs similar analysis on the involved image recognition results and the prior involved image recognition results. The prior involved image recognition results represent the accurate annotation of other image features and semantic information associated with the monitoring target in the image, while the involved image recognition results are the recognition conclusions drawn by the involved image recognition neural network based on the third monitoring implicit representation. Similarly, the computer system uses appropriate methods to measure the difference between the two. For example, in an identification task involving the layout of equipment in a high-altitude work scene, the prior involved image recognition results indicate that a certain specific construction equipment is located in the upper left corner of the work area, while the involved image recognition results identify it as being located in the upper right corner.

[0079] After calculating the difference between the monitoring target state recognition result and the prior monitoring target state, and the difference between the involved image recognition result and the prior involved image recognition result, the computer system needs to fuse these two difference measurements to obtain a comprehensive adjustment cost value. There are many fusion methods, such as weighted summation, which will not be described here.

[0080] The calibration cost value comprehensively reflects the degree of difference between the high-altitude operation status monitoring neural network and the expected level in terms of monitoring target status recognition and involved image recognition. Through this value, the computer system can clearly understand the current performance of the network, and then adjust the network parameters of the first branch network component and the second branch network component according to this calibration cost value to optimize the performance of the network, so that it can more accurately identify the monitoring target status and involved image information in subsequent processing, and improve the reliability and accuracy of the entire high-altitude operation status monitoring system. By continuously repeating this process, that is, acquiring new image data, calculating the calibration cost value, and adjusting the network parameters, the high-altitude operation status monitoring neural network can gradually learn more accurate patterns and features, thereby providing more reliable status monitoring services for high-altitude operations in practical applications.

[0081] Step S60: Based on the adjustment cost value, adjust the network parameters of the first branch network component and the second branch network component.

[0082] The computer system obtains the calibration cost value calculated in step S50. The calibration cost value is obtained by integrating the first sub-calibration cost value generated by comparing the monitoring target state recognition result with the prior monitoring target state, and the second sub-calibration cost value generated by comparing the involved image recognition result with the prior involved image recognition result, and is obtained by fusion (e.g., weighted summation). This value reflects the degree of difference between the output result of the current aerial work state monitoring neural network and the expected result.

[0083] Next, the computer system uses an optimization algorithm to adjust the network parameters of the first branch network component and the second branch network component based on the adjustment cost value. Network parameters are adjustable parameters in the neural network, such as weights and biases. These parameters determine how the neural network processes and transforms input data, thereby affecting the output results of the network.

[0084] In actual operation, optimization algorithms such as stochastic gradient descent (SGD) and its variants Adagrad, Adadelta, Adam, etc. can be used to adjust network parameters.

[0085] For the second branch network component, the computer system will also adjust its network parameters according to the principle of the above optimization algorithm. During the adjustment process, the computer system repeats the process of updating the network parameters based on the adjustment cost value many times until a certain stop condition is met. The stop condition can be that the adjustment cost value drops below a predetermined threshold, or after a certain number of iterations, the adjustment cost value no longer decreases significantly.

[0086] By adjusting the network parameters of the first branch network component and the second branch network component based on the calibration cost value, the computer system can gradually optimize the high-altitude work status monitoring neural network so that when faced with actual high-altitude work sensor monitoring images, it can more accurately identify the state relationship of the monitoring target and perform related involved image recognition, thereby improving the accuracy of work status detection and providing more reliable support for the safety of high-altitude operations.

[0087] As an implementation manner, the above step S300, based on the first monitoring implicit representation, performs monitoring target state recognition on the first operation sensor monitoring image to obtain the monitoring target state recognition result, including:

[0088] Step S310: performing feature comparison on the first operation sensor monitoring image in a preset feature library to obtain a second monitoring implicit representation corresponding to the first operation sensor monitoring image;

[0089] Step S320: implicitly integrating the first monitoring implicit representation and the second monitoring implicit representation to obtain a fourth monitoring implicit representation;

[0090] Step S330: Load the fourth monitoring implicit representation into the second branch network component, perform monitoring target state recognition on the first operation sensor monitoring image, and obtain the monitoring target state recognition result.

[0091] In step S310, the computer system performs feature comparison on the first operation sensor monitoring image in the preset feature library to obtain the second monitoring implicit representation corresponding to the image. The preset feature library is a database that stores a variety of typical image features. These features are carefully extracted and organized to cover the feature patterns that may be presented in different scenes and different monitoring targets. Feature comparison is a process of matching and similarity calculating the features of the first operation sensor monitoring image with the existing features in the feature library.

[0092] In the specific implementation, the computer system will first extract features from the first operation sensor monitoring image. A variety of feature extraction algorithms can be used, such as the scale-invariant feature transform (SIFT) algorithm. The SIFT algorithm can extract key points in the image and calculate its descriptors, which are a quantitative representation of the local features of the image. For a first operation sensor monitoring image I of size M×N, the computer system can obtain a series of key points k through the SIFT algorithm. i and its corresponding descriptor d i , where i=1, 2,…, n, n is the number of key points.

[0093] Next, the computer system compares these extracted features with the features in the preset feature library. Assume that there are m feature templates T in the feature library. j , j = 1, 2, ..., m, each feature template also has a corresponding descriptor. The computer system calculates the descriptor d of the first task sensor monitoring image i With each feature template descriptor T j The commonly used similarity measurement method is to calculate the Euclidean distance between them.

[0094] The computer system performs such calculations on the descriptors of each key point, and then selects matching results with higher similarity according to the set threshold. For example, when the Euclidean distance is less than a certain threshold, it is considered that a match is found. Through this matching information, the computer system can construct the correspondence between the first operation sensor monitoring image and the features in the feature library, thereby obtaining the second monitoring implicit representation.

[0095] For example, suppose that in the first operation sensor monitoring image of a high-altitude operation scene, 100 key points and their descriptors are extracted through the SIFT algorithm. In the preset feature library, there are feature templates for monitoring targets such as seat belt buckles and safety helmets. After the computer system calculates the Euclidean distance between these key point descriptors and the descriptors in the feature library, it is found that the descriptors of 20 key points have a high similarity with the descriptors of the seat belt buckle feature template, meeting the threshold requirement. Based on this matching information, the computer system can construct a vector, each dimension of which represents different matching feature information. This vector is the second monitoring implicit representation.

[0096] In step S320, the computer system implicitly integrates the first monitoring implicit representation and the second monitoring implicit representation to obtain a fourth monitoring implicit representation. The first monitoring implicit representation is the result obtained after the first branch network component performs data feature representation on the first operation sensor monitoring image, which includes the original features of the image and the abstract features learned by the neural network. The second monitoring implicit representation is obtained by comparing with the preset feature library, reflecting the matching relationship between the image and the known feature pattern.

[0097] When implementing implicit representation integration, the computer system can use a variety of methods, such as weighted summation. Assume that the first monitoring implicit representation is a vector , the second monitoring is implicitly represented as a vector , and their dimensions are First, the computer system performs dimension alignment on the two vectors, converting them into vectors of the same dimension r by interpolation, dimensionality reduction, etc. .

[0098] The computer system then assigns weights to the two implicit representations based on their importance. Assume that the weight of the first monitored implicit representation is , the weight of the second monitoring implicit representation is ,and The fourth monitoring implicit representation It can be calculated by the following formula:

[0099] ;

[0100] For example, suppose the first monitoring implicitly represents is a vector of dimension 50, the second monitoring implicit representation is a vector of dimension 30. The computer system interpolates Expanded to a vector of dimension 50 After analysis, it is considered that the first monitoring implicit representation is relatively more important in this scenario, and the weight is assigned Then the fourth monitoring implicitly indicates The value of each dimension of is .

[0101] Another integration method is to use neural network fusion. The computer system can build a small fusion neural network, take the first monitoring implicit representation and the second monitoring implicit representation as input, and output the fourth monitoring implicit representation after learning and transformation by the network. This fusion neural network can contain multiple hidden layers and be trained by the back propagation algorithm to learn how to effectively integrate the two implicit representations.

[0102] In practical applications, for aerial work images, the first monitoring implicit representation may contain underlying features such as the spatial position and color of the object in the image, while the second monitoring implicit representation may contain information matching the features of the safety equipment. By integrating these two implicit representations, the fourth monitoring implicit representation can more comprehensively reflect the features of the monitoring target in the image.

[0103] Finally, in step S330, the computer system loads the fourth monitoring implicit representation into the second branch network component, performs monitoring target state recognition on the first operation sensor monitoring image, and obtains the monitoring target state recognition result. The second branch network component is a neural network module specifically used for image monitoring target state recognition. After training and learning, it can analyze and judge the input feature representation and output the state relationship of the monitoring target.

[0104] The second branch network component usually includes multiple hidden layers and an output layer. When the computer system inputs the fourth monitoring implicit representation into the second branch network component, the data first enters the input layer and then is processed by each hidden layer in turn. In the hidden layer, the data will be calculated and transformed by neurons, which perform nonlinear transformations on the input data through weights and activation functions. After layer-by-layer transformations of multiple hidden layers, the data finally reaches the output layer. The output layer outputs the monitoring target state recognition result based on the output results of the hidden layer through the calculation of the weight matrix and the bias vector. Assume that the weight matrix of the output layer is , the bias vector is , then the monitoring target state recognition result It can be calculated by the following formula:

[0105] ;

[0106] in, is the output of the last hidden layer.

[0107] The monitoring target state recognition result y is a vector whose dimensions correspond to different monitoring target state categories. For example, for high-altitude work scenes, there may be different state categories such as "safety belt worn normally" and "safety helmet not worn". The value of each dimension of vector y represents the probability that the monitoring target in the image is in the corresponding state category. The computer system can compare the values ​​of each dimension of vector y and select the state category corresponding to the dimension with the largest probability as the final monitoring target state recognition result.

[0108] For example, suppose the output vector y = [0.2, 0.8, 0.1] corresponds to the three categories of "safety belt worn normally", "safety helmet not worn", and "other states". Since the value of the second dimension, 0.8, is the largest, the computer system determines that the state of the monitoring target in the first operation sensor monitoring image is "safety helmet not worn".

[0109] As an implementation manner, the first monitoring implicit representation includes a pixel implicit representation, and the second monitoring implicit representation includes a first image block implicit representation; step S320, implicitly integrating the first monitoring implicit representation and the second monitoring implicit representation to obtain a fourth monitoring implicit representation, including:

[0110] Step S321: performing image segmentation on the first operation sensor monitoring image;

[0111] Step S322: Based on the result obtained by the image segmentation, the implicit representation of the pixel is integrated to obtain an implicit representation of a second image block corresponding to the monitoring target in the first operation sensor monitoring image;

[0112] Step S323: implicitly integrate the first image block implicit representation and the second image block implicit representation to obtain a fourth monitoring implicit representation.

[0113] In step S321, the computer system performs image segmentation on the first operation sensor monitoring image. Image segmentation is the process of dividing an image into multiple regions with similar features, and its purpose is to separate the object or region of interest in the image from the background for more accurate subsequent analysis and processing.

[0114] Computer systems can use a variety of image segmentation techniques, such as threshold-based segmentation methods. This method sets one or more thresholds and compares the pixel values ​​of the image with the thresholds to divide the pixels into different categories. For a grayscale image I(x, y), where x and y represent the horizontal and vertical coordinates of the pixels, respectively, assuming that the threshold is set to T, the threshold-based segmentation can be expressed as:

[0115] ; Where G(x, y) is the binary image after segmentation, where pixels with a value of 1 constitute one region and pixels with a value of 0 constitute another region. For example, in a grayscale image of aerial work, the grayscale value of the background is generally low, while the grayscale value of the operator and related safety equipment is high. The computer system can segment the image into two parts: the foreground (operators and safety equipment) and the background by selecting an appropriate threshold (such as using the Otsu method to automatically calculate the threshold).

[0116] Another commonly used image segmentation method is region-based segmentation. This method starts with one or more seed points in the image and merges adjacent pixels with similar features to the seed points into the same region based on certain similarity criteria. For example, based on the region growing algorithm, one or more seed points (x0, y0) are first selected, and then a similarity metric function S(p, q) is defined to measure the similarity between two pixels p and q. In each iteration, the computer system checks the neighboring pixel q of the seed point. If S((x0, y0), q) meets certain conditions (such as the similarity is greater than a certain threshold), q is merged into the current region. This process is repeated until there are no new pixels to merge.

[0117] In addition, image segmentation based on edge detection is also a common method. The edge is where the grayscale value in the image changes sharply, representing the boundary of the object. Computer systems can use various edge detection operators, such as the Sobel operator and the Canny operator, to detect the edges in the image. Taking the Sobel operator as an example, it detects the edge by calculating the gradient of the image in the horizontal and vertical directions. For an image I(x, y), the horizontal gradient G x and the vertical gradient G y It can be calculated by the following formula:

[0118] ;

[0119] ;

[0120] in represents the convolution operation. Then, the magnitude of the gradient is calculated and direction , the computer system can determine the edge locations in the image. Based on this edge information, the image is segmented into different regions.

[0121] In the high-altitude work image segmentation, the computer system uses appropriate image segmentation methods to segment different parts of the image, such as workers, safety equipment (such as helmets, safety belts, etc.), work tools, and background, providing a basis for the subsequent integration of implicit pixel representation.

[0122] In step S322, the computer system integrates the implicit representation of the pixel based on the result of the image segmentation to obtain the implicit representation of the second image block corresponding to the monitoring target in the first operation sensor monitoring image. The pixel implicit representation is part of the first monitoring implicit representation, which contains the feature information of each pixel.

[0123] When performing the integration, the computer system considers the feature similarity of the pixels in each segmented region. For each segmented region, the computer system can calculate the statistical features implicitly represented by the pixels in the region, such as the mean, variance, etc. Assume that there are N pixels in a segmented region R, and its pixels are implicitly represented as vectors , i = 1, 2,…, N. The mean vector of the region It can be calculated by the following formula:

[0124] ;

[0125] This mean vector It can be regarded as an integrated result of the implicit representation of the pixels in the area, representing the overall characteristics of the area. The computer system performs such calculations on each segmented area to obtain the feature vectors of each area.

[0126] The computer system then merges or further processes the adjacent regions based on the similarity between the feature vectors of these regions. For example, if the Euclidean distance between the feature vectors of two adjacent regions is less than a certain threshold, the two regions can be merged into a larger region and the feature vector of the merged region can be recalculated.

[0127] In this way, the computer system integrates the implicit representations of the pixels in each segmented area in the image to form an implicit representation of the second image block corresponding to different monitoring targets or target parts. For example, in the high-altitude work image, the head area, body area, and helmet area of ​​the operator are obtained through image segmentation. After the computer system integrates the implicit representations of the pixels in these areas, it obtains the implicit representations of the second image blocks representing the head, body, and helmet of the operator respectively. These implicit representations contain more representative feature information of each area, providing more effective data for the subsequent integration with the implicit representation of the first image block.

[0128] In step S323, the computer system implicitly integrates the first image block implicit representation and the second image block implicit representation to obtain a fourth monitoring implicit representation. The first image block implicit representation comes from the second monitoring implicit representation, which is based on the image block feature representation obtained by comparing with a preset feature library.

[0129] The computer system can use a variety of methods to integrate the implicit representations of these two image blocks. One method is the splicing method, which is to splice the implicit representation of the first image block and the implicit representation of the second image block in the vector dimension. For example, the implicit representation of the first image block is a vector with a dimension of 30, which represents the information that matches the helmet feature in the preset feature library; the implicit representation of the second image block is a vector with a dimension of 20, which represents the feature information of the helmet area obtained from the integration of image segmentation and pixel implicit representation. Through splicing, a fourth monitoring implicit representation vector with a dimension of 50 is obtained. This vector integrates information from two aspects and more comprehensively describes the characteristics of the helmet in the image.

[0130] Another integration method is weighted fusion. The computer system assigns weights to the implicit representation of the first image block and the implicit representation of the second image block according to their importance. For example, in a certain high-altitude work scene, after many experiments and analyses, the computer system determined that the implicit representation of the first image block is relatively more important in describing the characteristics of the seat belt, with a weight w1 = 0.6, and the implicit representation of the second image block has a weight w2 = 0.4. Through weighted fusion, the fourth monitoring implicit representation obtained can better combine the advantages of the two representations and accurately reflect the characteristics of the seat belt in the image.

[0131] In addition, the computer system can also use a neural network fusion method. A special fusion neural network is constructed, and the implicit representation of the first image block and the implicit representation of the second image block are used as inputs. After learning and transformation by the network, the fourth monitoring implicit representation is output. This fusion neural network can be trained through a back propagation algorithm to learn how to adjust the weights according to different inputs to achieve the best fusion effect.

[0132] Through a series of operations from step S321 to step S323, the computer system starts with the segmentation of the first operation sensor monitoring image, gradually integrates the implicit representation information at different levels, and finally obtains the fourth monitoring implicit representation. This process makes full use of the spatial structure information of the image and the feature representations from different sources, so that the fourth monitoring implicit representation can more comprehensively and accurately describe the characteristics of the monitoring target in the image, and provides high-quality feature input for subsequent loading to the second branch network component for monitoring target state identification, which helps to improve the accuracy and reliability of monitoring target state identification, thereby better ensuring the safety of high-altitude operations. In practical applications, different image segmentation and implicit representation integration methods can be selected and adjusted according to specific high-altitude operation scenarios and monitoring needs to achieve the best monitoring effect.

[0133] As an implementation manner, the involved image recognition neural network includes a monitoring target classification network; the step S40, loading the third monitoring implicit representation into the involved image recognition neural network, performing involved image recognition processing on the second operation sensor monitoring image, and obtaining an involved image recognition result corresponding to the second operation sensor monitoring image, includes:

[0134] Step S41: loading the third monitoring implicit representation into the monitoring target classification network, inferring the target classification result of the second operation sensor monitoring image, and obtaining the target classification inference result corresponding to the second operation sensor monitoring image.

[0135] The monitoring target classification network is a trained neural network model designed to analyze and classify the input feature representation to determine the category of the monitoring target in the image. The network is built based on a deep learning algorithm, usually containing multiple hidden layers and an output layer, and can automatically learn complex patterns and features in the input data.

[0136] The computer system first inputs the third monitoring implicit representation into the input layer of the monitoring target classification network. The third monitoring implicit representation is the result obtained after the second operation sensor monitoring image is represented by the first branch network component for data feature representation. It contains rich feature information of the monitoring target in the image. For example, in a high-altitude operation scene, the second operation sensor monitoring image may contain monitoring targets such as operators, safety helmets, and safety belts. The third monitoring implicit representation is the representation form after the features of these targets are extracted and encoded. It may be a vector or tensor, each dimension of which represents a different feature dimension.

[0137] When the third monitoring implicit representation enters the monitoring target classification network, it will be processed by each hidden layer in turn. In the hidden layer, the neurons perform nonlinear transformation on the input data through the weight matrix and activation function.

[0138] After multiple hidden layer transformations, the data finally reaches the output layer. The output layer outputs the target classification inference result corresponding to the second operation sensor monitoring image based on the output result of the hidden layer through the calculation of the weight matrix and the bias vector.

[0139] For example, suppose that after the monitoring target classification network is trained, it is used to infer a second operation sensor monitoring image containing a worker and a helmet. After the third monitoring implicit representation is input into the network, it is processed by the hidden layer and finally obtains the target classification inference result vector y= [0.8, 0.1, 0.05, 0.05] at the output layer. The first dimension of this vector corresponds to the "safety helmet" category, with a value of 0.8, indicating that the computer system believes that the probability that the monitoring target in the image is a safety helmet is 80% based on the inference of the monitoring target classification network; the second dimension corresponds to the "safety belt" category, with a value of 0.1, indicating that the probability that the monitoring target in the image is a safety belt is 10%; the third dimension corresponds to the "operating tool" category, with a value of 0.05, indicating a probability of 5%; the fourth dimension corresponds to the "worker body part" category, with the same value of 0.05, indicating a probability of 5%. Based on this inference result, the computer system can determine that the main monitoring target in the image is a safety helmet.

[0140] In the actual implementation process, the monitoring target classification network needs to be trained in advance. The training process usually uses a large amount of annotated second-job sensor monitoring image data, and the annotation information clearly indicates the category of the monitoring target in the image. Through the back-propagation algorithm, the network continuously adjusts the weights and biases of the network based on the difference between the predicted results and the actual annotations (for example, the cross entropy loss function measures this difference), so that the network can accurately classify different categories of monitoring targets.

[0141] By loading the third monitoring implicit representation into the monitoring target classification network for reasoning, the computer system can automatically identify the category of the monitoring target in the second operation sensor monitoring image, providing a basis for subsequent more in-depth analysis and processing based on these classification results, thereby better realizing the monitoring and evaluation of the high-altitude operation status.

[0142] As another embodiment, the involved image recognition neural network includes a monitoring target commonality measurement network; the step S40, loading the third monitoring implicit representation into the involved image recognition neural network, performing involved image recognition processing on the second operation sensor monitoring image, and obtaining an involved image recognition result corresponding to the second operation sensor monitoring image, includes:

[0143] Step S401: arbitrarily segmenting the second operation sensor monitoring image to obtain a first target operation sensor monitoring image and a second target operation sensor monitoring image;

[0144] Step S402: determining, in the third monitoring implicit representation, a first target implicit representation obtained based on the first target operation sensor monitoring image processing, and a second target implicit representation obtained based on the second target operation sensor monitoring image processing;

[0145] Step S403: Load the first target implicit representation and the second target implicit representation into the monitoring target commonality measurement network, measure the image commonality degree of the first target operation sensor monitoring image and the second target operation sensor monitoring image, and obtain the image commonality measurement result of the first target operation sensor monitoring image and the second target operation sensor monitoring image.

[0146] In step S401, the computer system arbitrarily segments the second operation sensor monitoring image to obtain a first target operation sensor monitoring image and a second target operation sensor monitoring image. Image segmentation is an operation that divides a complete image into multiple sub-images with specific meanings, and its purpose is to provide a basis for subsequent analysis of the relationship between different parts.

[0147] The computer system can use a variety of image segmentation techniques to implement this step. For example, the method based on random region division randomly generates some boundary points in the image, and then divides the image into different regions according to these boundary points. Assuming that the size of the second operation sensor monitoring image is M×N pixels, the computer system can generate a series of points (xi, yi) within the coordinate range of the image through a random number generator, where i = 1, 2,…, k, and k is the number of generated points. These points are used as boundary points to divide the image into multiple regions, and then any two regions are selected from them as the first target operation sensor monitoring image and the second target operation sensor monitoring image.

[0148] Another method is based on semantic segmentation technology. Semantic segmentation aims to classify each pixel in the image into different semantic categories, such as "worker", "safety equipment", "background", etc. The computer system can use semantic segmentation models in deep learning, such as U-Net, Mask R-CNN, etc. Taking the U-Net model as an example, it is a neural network with an encoder-decoder structure. The encoder part gradually reduces the resolution of the image through convolutional layers and pooling layers to extract the features of the image; the decoder part gradually restores the resolution of the image through deconvolution layers and upsampling operations, and classifies each pixel based on the feature information of the encoder. Through the semantic segmentation model, the computer system can segment the second operation sensor monitoring image into regions of different semantic categories, and then select two representative regions from these regions, which are defined as the first target operation sensor monitoring image and the second target operation sensor monitoring image. For example, in a second operation sensor monitoring image of a high-altitude operation, through the semantic segmentation model, the computer system segments the image into the operator's body area, the helmet area, the safety belt area, and the background area. The computer system selects the helmet area as the first target operation sensor monitoring image and the safety belt area as the second target operation sensor monitoring image, so as to subsequently analyze the relationship between these two areas closely related to safety.

[0149] In step S402, the computer system determines in the third monitoring implicit representation the first target implicit representation obtained based on the first target operation sensor monitoring image processing, and the second target implicit representation obtained based on the second target operation sensor monitoring image processing. The third monitoring implicit representation is the result obtained after the second operation sensor monitoring image is represented by the first branch network component for data features, and it contains rich feature information of the entire image.

[0150] Since the first branch network component extracts and transforms features of the image, the third monitoring implicit representation contains feature information corresponding to each part of the image. The computer system locates and extracts the corresponding parts in the third monitoring implicit representation according to the position and range information of the first target operation sensor monitoring image and the second target operation sensor monitoring image in the original second operation sensor monitoring image, and obtains the first target implicit representation and the second target implicit representation respectively.

[0151] For example, assume that the third monitoring implicit representation is a multidimensional vector whose dimensions are related to the spatial and feature dimensions of the second operation sensor monitoring image. The position of the first target operation sensor monitoring image in the second operation sensor monitoring image corresponds to a subvector range in the third monitoring implicit representation. The computer system extracts this subvector from the third monitoring implicit representation as the first target implicit representation by means of coordinate mapping and feature matching. Similarly, for the second target operation sensor monitoring image, similar operations are performed to obtain the second target implicit representation.

[0152] In specific implementation, the computer system can be realized by recording the coordinate information during image segmentation and the feature mapping relationship during the processing of the first branch network component. For example, when performing image segmentation, the coordinates of the upper left corner and the lower right corner of the first target operation sensor monitoring image and the second target operation sensor monitoring image are recorded. When processing the image, the first branch network component establishes a mapping relationship between the image coordinates and the implicit representation vector position. Based on these mapping relationships, the computer system accurately extracts the first target implicit representation and the second target implicit representation corresponding to the first target operation sensor monitoring image and the second target operation sensor monitoring image in the third monitoring implicit representation.

[0153] In step S403, the computer system loads the first target implicit representation and the second target implicit representation into the monitoring target commonality measurement network, measures the image commonality degree of the first target operation sensor monitoring image and the second target operation sensor monitoring image, and obtains the image commonality measurement result of the first target operation sensor monitoring image and the second target operation sensor monitoring image. The monitoring target commonality measurement network is a neural network specially designed to measure the similarity or commonality degree between two image feature representations.

[0154] The network usually contains multiple hidden layers and an output layer. When the first target implicit representation and the second target implicit representation are input into the monitoring target commonality measurement network, they first enter the input layer. In the hidden layer, the network will extract and fuse the features of the two implicit representations. For example, in a hidden layer, the features of the two implicit representations are fused by calculating the element-by-element product and element-by-element addition of the two implicit representations. Assuming that the first target implicit representation is vector a and the second target implicit representation is vector b, both with dimensions d, the following operations may be performed in the hidden layer:

[0155] ;

[0156] in represents the element-by-element product, c is the fused vector, which contains the comprehensive feature information of the implicit representation of the first target and the implicit representation of the second target.

[0157] After being processed by multiple hidden layers, the data reaches the output layer. The output layer outputs a scalar value by calculating the weight matrix and the bias vector. This value is the image commonality measurement result of the first target operation sensor monitoring image and the second target operation sensor monitoring image. Assuming that the weight vector of the output layer is w, the bias is b, and the output vector of the hidden layer is c, the image commonality measurement result s can be calculated by the following formula: s=w T c+b; the size of the image commonality measurement result s reflects the degree of commonality between the first target operation sensor monitoring image and the second target operation sensor monitoring image. The larger the value, the more similar the two images are in terms of features; the smaller the value, the greater their difference.

[0158] For example, in the aerial work scene, the first target work sensor monitoring image is the image representation of the helmet area, and the second target work sensor monitoring image is the image representation of the safety belt area. After inputting their corresponding first target implicit representation and second target implicit representation into the monitoring target commonality measurement network, the image commonality measurement result obtained by the network calculation is 0.3. This shows that the degree of commonality in features between the helmet area and the safety belt area is relatively low, because they belong to different types of safety equipment with different appearance and functional characteristics.

[0159] If the first target operation sensor monitoring image is an image representation of the left hand of an operator, and the second target operation sensor monitoring image is an image representation of the right hand of the operator, after being measured by the monitoring target commonality measurement network, the image commonality measurement result may be 0.8. This shows that the left hand and the right hand have high similarity in image features, because they are both limbs of the human body and have similar characteristics such as color, texture and shape. In practical applications, the monitoring target commonality measurement network needs to be trained in advance. The training process usually uses a large amount of paired image data, which have different degrees of commonality and corresponding annotation information. The annotation information represents the true value of their commonality degree. Through the back propagation algorithm, the network continuously adjusts the weights and biases of the network according to the difference between the predicted commonality measurement result and the true annotation (for example, the mean square error loss function measures this difference), so that the network can accurately measure the commonality degree between different images.

[0160] Through a series of operations from step S401 to S403, the computer system can deeply analyze the relationship between different parts in the second operation sensor monitoring image, and the image commonality measurement results obtained by the monitoring target commonality measurement network provide an additional information dimension for high-altitude operation status monitoring. This information can help determine the correlation between different elements in the operation process, such as the coordinated use of safety equipment, the relationship between the operator's body parts and the surrounding environment, etc., so as to more comprehensively and accurately evaluate the status of high-altitude operations and provide strong support for ensuring the safety of high-altitude operations. In the actual implementation process, different image segmentation methods, implicit representation extraction methods, and the design of the monitoring target commonality measurement network can be adjusted and optimized according to the specific high-altitude operation scene and monitoring needs to obtain more accurate and reliable involved image recognition results.

[0161] As an implementation manner, the step S50, based on the monitoring target state recognition result and the prior monitoring target state, and the involved image recognition result and the prior involved image recognition result, obtains the adjusted adjustment cost value, including:

[0162] Step S51: determining a first sub-adjustment cost value for performing monitoring target state recognition on the second operation sensor monitoring image based on the monitoring target state recognition result and the prior monitoring target state;

[0163] Step S52: determining a second sub-calibration cost value of performing image recognition processing on the second operation sensor monitoring image based on the image recognition result and the prior image recognition result;

[0164] Step S53: merging the first sub-adjustment cost value and the second sub-adjustment cost value to obtain an adjusted adjustment cost value.

[0165] In step S51, the computer system determines the first sub-adjustment cost value for monitoring target state recognition of the second operation sensor monitoring image based on the monitoring target state recognition result and the prior monitoring target state. The monitoring target state recognition result is the judgment on the monitoring target state obtained by the aerial work state monitoring neural network after analyzing the second operation sensor monitoring image, and the prior monitoring target state is the known and accurate actual state information of the monitoring target in the image.

[0166] The computer system measures the difference between the two by selecting an appropriate loss function to determine the first sub-adjustment cost. Commonly used loss functions such as the cross entropy loss function have good effects on multi-classification problems. Assuming that there are C categories of monitoring target states, the monitoring target state recognition result can be expressed as a probability distribution vector ,in Represents the predicted probability that the monitoring target belongs to the i-th category; the prior monitoring target state can be represented as a one-hot encoding vector , where only the dimension value corresponding to the actual category is 1, and the other dimension values ​​are 0. Cross entropy loss function The calculation formula is:

[0167] ;

[0168] For example, in a high-altitude operation scenario, the monitoring target state categories may be "safety belt worn normally", "safety belt not worn", "safety helmet worn normally", "safety helmet not worn", etc. For a second operation sensor monitoring image, the monitoring target state recognition result is p = [0.2, 0.3, 0.4, 0.1], and the prior monitoring target state indicates that the actual situation is "safety helmet not worn", and the corresponding one-hot encoding vector q = [0, 0, 0, 1]. The computer system calculates the above cross entropy loss function:

[0169] ;

[0170] This value of 2.30 is the first sub-adjustment cost value of the image in terms of monitoring target state recognition. Through this calculation method, the computer system can quantify the degree of difference between the monitoring target state recognition result and the prior monitoring target state. The larger the first sub-adjustment cost value, the greater the deviation between the recognition result and the actual situation, and more adjustments need to be made to the neural network to improve its accuracy.

[0171] Another loss function that can be used to calculate the first sub-tuning cost value is the mean square error (MSE) loss function, which is suitable for continuous value state prediction or state problems that can be converted to continuous value representation.

[0172] In step S52, the computer system determines a second sub-calibration cost value for performing involved image recognition processing on the second operation sensor monitoring image based on the involved image recognition result and the prior involved image recognition result. The involved image recognition result is the result obtained by loading the third monitoring implicit representation into the involved image recognition neural network to process the second operation sensor monitoring image, and the prior involved image recognition result is the accurate involved image recognition information about the image known in advance.

[0173] Similarly, the computer system selects an appropriate method to quantify the difference between the two. If the image recognition result is a classification result, such as classifying objects in the image into different sets of categories, the computer system can use the classification error rate to calculate the difference. Assuming that the number of correctly classified samples in the image recognition result is m and the total number of samples is N, the classification error rate The calculation formula for the second sub-tuning cost value is:

[0174] ;

[0175] For example, when performing image recognition on a second-operation sensor monitoring image, the prior image recognition results show that the objects in the image should be accurately classified into 3 category sets, and a total of 10 object samples are recognized. However, only 7 objects in the image recognition results are correctly classified into the corresponding category sets. Then according to the above formula, the classification error rate is 0.3, which is the second sub-calibration cost value of the image in image recognition.

[0176] If the image recognition result is a similarity score or other continuous value representation, the computer system can use a loss function such as absolute error or mean square error to calculate the difference. For example, the absolute error loss function The formula used to calculate the difference between the involved image recognition result r and the prior involved image recognition result s is:

[0177] ;

[0178] Assume that the result of the involved image recognition is 0.6, which indicates the similarity score between the image and a reference image, and the prior involved image recognition result is 0.8, and the absolute error loss function is used to calculate 0.2, which is the corresponding second sub-adjustment cost value. In this way, the computer system can effectively calculate the second sub-adjustment cost value for different forms of involved image recognition results and prior information, reflecting the accuracy of involved image recognition.

[0179] In step S53, the computer system fuses the first sub-adjustment cost value and the second sub-adjustment cost value to obtain the adjusted adjustment cost value. The fusion method usually adopts the weighted summation method, which can assign different weights to different sub-adjustment cost values ​​according to their importance. The determination of weights can be achieved in many ways. One is to manually set it based on experience and domain knowledge. For example, according to the focus of high-altitude work safety monitoring, it is considered that the identification of the monitoring target state is more critical, so as to give the first sub-adjustment cost value a larger weight. Another method is to use a data-driven approach, such as using cross-validation technology to try different weight combinations on the training data, calculate the corresponding adjustment cost values, and select the optimal weight combination based on the performance on the verification set.

[0180] Through a series of operations from step S51 to S53, the computer system accurately quantifies the difference between the high-altitude operation status monitoring neural network in terms of monitoring target status recognition and involved image recognition and the expected results, and obtains the calibration cost value through a reasonable fusion method. This calibration cost value provides a clear basis for the subsequent adjustment of the network parameters of the first branch network component and the second branch network component, so that the neural network can make targeted adjustments based on the deviation between the actual performance and the expected, and gradually optimize its performance, thereby improving the ability to understand the high-altitude operation images and the accuracy of status detection, and providing reliable technical support for ensuring the safety of high-altitude operations. In actual applications, the computer system repeats this process for a large number of second operation sensor monitoring images, continuously accumulating and analyzing the calibration cost values, so as to achieve continuous optimization of the neural network and adapt to different high-altitude operation scenarios and monitoring needs.

[0181] As an embodiment, the method further includes:

[0182] Step S70: determining whether each of the second operation sensor monitoring images omits a corresponding a priori monitoring target state or a priori involved image recognition result;

[0183] Step S80: If the second operation sensor monitoring image omits the corresponding prior monitoring target state, skip the step of loading the third monitoring implicit representation into the second branch network component and identifying the monitoring target state of the second operation sensor monitoring image.

[0184] Step S90: If the second operation sensor monitoring image omits the corresponding prior involved image recognition result, skip the step of loading the third monitoring implicit representation into the involved image recognition neural network and performing involved image recognition processing on the second operation sensor monitoring image.

[0185] In step S70, the computer system determines whether each second operation sensor monitoring image omits the corresponding prior monitoring target state or prior involved image recognition result. The prior monitoring target state and the prior involved image recognition result are important references for training and evaluating the high-altitude operation state monitoring neural network. The computer system needs to check each second operation sensor monitoring image one by one to determine whether it contains complete prior information.

[0186] In the implementation process, the computer system can achieve this goal by establishing a data index and information recording mechanism. For example, the file names or identifiers of all second operation sensor monitoring images are associated and stored with their corresponding prior monitoring target states and prior involved image recognition results. During the inspection, the computer system traverses the recorded information of all images and compares each image to see whether there is corresponding prior information. If the record of the prior monitoring target state or the prior involved image recognition result of a certain image is empty, then the computer system determines that there is information missing in the image.

[0187] Taking the aerial work scene as an example, suppose there are 100 second work sensor monitoring images in the computer system for training the neural network. For each image, the corresponding prior monitoring target state should be clearly marked with status information such as "safety belt worn normally and safety helmet worn normally", "safety belt not worn but safety helmet worn normally", etc. The prior involved image recognition results may include semantic analysis results related to safe work of each object in the image, such as "the placement of work tools in the image complies with safety regulations". When the computer system checked, it was found that the prior monitoring target state record of the 15th image was empty and the prior involved image recognition result of the 37th image was missing, so these two images were identified as having information omission problems.

[0188] Entering step S80, if the second operation sensor monitoring image omits the corresponding prior monitoring target state, the computer system skips the step of loading the third monitoring implicit representation into the second branch network component and identifying the monitoring target state of the second operation sensor monitoring image. This is because there is a lack of accurate prior monitoring target state as a reference, and the monitoring target state identification of the image may produce inaccurate results, and no effective comparison basis can be obtained when the adjustment cost value is calculated later, thereby affecting the adjustment effect of the neural network.

[0189] When executing this step, the computer system decides whether to skip the corresponding operation by judging the missing information detected in step S70. When it is detected that the prior monitoring target state of a certain image is missing, the computer system automatically marks the monitoring target state identification link of the image in the subsequent process as skipped. For example, for the 15th image mentioned above, since its prior monitoring target state is missing, the computer system no longer loads the third monitoring implicit representation obtained by the first branch network component of the image into the second branch network component for monitoring target state identification, but directly enters the processing flow of the next image. In this way, the computer system can effectively avoid invalid calculations due to the missing prior monitoring target state, save computing resources and improve processing efficiency.

[0190] In step S90, if the second operation sensor monitoring image omits the corresponding prior involved image recognition result, the computer system skips the step of loading the third monitoring implicit representation into the involved image recognition neural network and performing involved image recognition processing on the second operation sensor monitoring image. Similarly, the lack of prior involved image recognition results will lead to the inability to accurately evaluate the output results of the involved image recognition neural network, thereby affecting the overall adjustment cost value calculation and the optimization of the neural network.

[0191] The computer system decides whether to skip the operation based on the check result of step S70. When it is determined that the prior involved image recognition result of a certain image is missing, the step of inputting the third monitored implicit representation into the involved image recognition neural network is automatically skipped. For example, for the 37th image mentioned above, since its prior involved image recognition result is empty, the computer system no longer loads its third monitored implicit representation into the involved image recognition neural network for processing, but continues to process the next image.

[0192] The technical means to implement this step are similar to step S80, and the process is controlled by conditional judgment statements. In this way, the computer system can ensure effective image recognition processing when the data is complete, ensuring the reliability of the calculation results and the effectiveness of the neural network adjustment.

[0193] Starting from the data integrity check, the data missing problem is dealt with in a targeted manner to avoid calculation errors and invalid training caused by incomplete data. In the entire process of adjusting the high-altitude operation status monitoring neural network, it is ensured that every data involved in the calculation and training is valid, thereby improving the neural network's ability to understand and analyze high-altitude operation images, making the monitoring results more accurate and reliable, and providing a solid foundation for the safety of high-altitude operations.

[0194] Furthermore, these steps are particularly effective in large-scale aerial work data processing. As the amount of data increases, the data missing situation may become more complex and diverse. By strictly executing steps S70-S90, the computer system can efficiently filter out problematic data and perform reasonable processing. For example, in a data set containing thousands of second operation sensor monitoring images, there may be different proportions of images missing the prior monitoring target state or the prior involved image recognition results. After a comprehensive inspection through step S70, the computer system can accurately mark these missing images. Then, in steps S80 and S90, skip processing is performed for different types of missing situations, respectively, avoiding a large amount of invalid computing resource consumption. From the perspective of neural network calibration, these steps help improve the accuracy and stability of calibration. If calculation and training are forced in the absence of data, the neural network may learn the wrong pattern or feature, thereby affecting its performance in practical applications. By skipping the relevant calculation steps of missing data, it can ensure that the neural network is learned and optimized on the basis of valid data, making its parameter adjustment more reasonable and the monitoring of aerial work status more accurate.

[0195] In addition, these three steps can also be combined with other data preprocessing and quality control measures. For example, in the data collection stage, the collection and verification of prior information can be strengthened to ensure the initial quality of the data. In terms of data storage and management, a more complete database structure can be established to facilitate the computer system to check and manage data integrity. At the same time, for the case of missing data, more complex processing methods can be further explored, such as data filling technology, trying to infer the missing prior information through other related data or models, so as to make better use of data resources and improve the effect of high-altitude operation status monitoring.

[0196] In actual application scenarios, different high-altitude working environments and monitoring requirements may have different impacts on these three steps. For example, in some special high-altitude working scenarios with extremely high safety requirements, the requirements for data integrity may be more stringent, and any data missing may be regarded as a serious problem. At this time, the computer system can adopt a more conservative strategy when executing steps S70-S90, such as directly eliminating images with missing data to ensure the accuracy of neural network training and monitoring. In some scenarios with high data volume requirements and relatively difficult data acquisition, the processing strategy can be appropriately relaxed. While skipping the missing data calculation steps, relevant information can be recorded for further analysis and processing.

[0197] The computer system checks and processes the data integrity of the second operation sensor monitoring image through steps S70-S90, which plays a vital role in the calibration of the high-altitude operation status monitoring neural network. These steps not only improve the efficiency and accuracy of data processing, but also ensure the performance and reliability of the neural network, providing strong support for the safety monitoring of high-altitude operations.

[0198] As an embodiment, the method further includes:

[0199] Step S1: Acquire a plurality of third operation sensor monitoring images;

[0200] Step S2: performing a feature masking operation on each of the third operation sensor monitoring images to obtain an operation sensor monitoring image sample set;

[0201] Step S3: Pre-debugging the first branch network component based on the operation sensor monitoring image sample set and the third operation sensor monitoring image.

[0202] In step S1, the computer system obtains a plurality of third operation sensor monitoring images. These images are collected from the aerial operation site and include monitoring targets in various operation scenes, such as operating personnel, safety equipment, and operating environment. There are many ways to obtain images. The most common method is to shoot in real time through a camera installed on the aerial operation equipment, or to extract key frames as image samples at a certain time interval from an existing operation video.

[0203] For example, in a high-altitude operation scene of a building construction, the computer system collects images at a frequency of one frame per minute through a high-definition camera installed on a tower crane for several hours, and obtains a large number of third-party operation sensor monitoring images. These images cover the operation conditions in different time periods and different construction stages. Some images show operators performing material lifting operations, while others show workers wearing safety equipment working on scaffolding.

[0204] In practical applications, computer systems can use professional image acquisition equipment and software to complete this task. Image acquisition equipment needs to have high resolution and adapt to different lighting conditions to ensure that the quality of the acquired images is clear and accurately reflects the situation on the job site. The acquired image data will be transmitted and stored in the storage device of the computer system for subsequent processing.

[0205] Entering step S2, the computer system performs a feature masking operation on each third operation sensor monitoring image to obtain an operation sensor monitoring image sample set. Feature masking operation is a data enhancement technology that aims to force the neural network to learn other more critical features in the image by randomly or specifically masking some features in the image, thereby improving the generalization ability and robustness of the network.

[0206] The computer system can perform feature shielding operations in a variety of ways. One of the ways is to perform random feature shielding operations on the third task sensor monitoring image. In this way, the computer system randomly selects some pixels or areas in the image and sets their feature values ​​to 0 or other specific values, thereby shielding these features. For example, for an image I of size M×N, the computer system can generate a series of random points (xi, yi) within the coordinate range of the image through a random number generator, where i = 1, 2,…, k, and k is the number of random points generated. Then, the pixel values ​​of these points and the surrounding pixel values ​​within a certain range are set to 0 to shield the local features of the image.

[0207] Another way is to perform feature shielding operations on multiple image blocks in the third operation sensor monitoring image. The computer system first divides the image into multiple image blocks of equal or different sizes, and then randomly selects some image blocks to perform feature shielding on these selected image blocks. For example, an image is divided into m×n image blocks of size a×b, and several of the image blocks are selected by random numbers, and all pixel values ​​of these image blocks are set to a fixed value (such as 0), so that the neural network cannot directly obtain feature information from these shielded image blocks, thereby prompting it to learn more representative features from other unshielded image blocks. Another way is to perform feature shielding operations on multiple monitoring targets in the third operation sensor monitoring image. The computer system first identifies the monitoring targets in the image, such as safety helmets, safety belts, work tools, etc., and then randomly selects some monitoring targets and performs feature shielding on the corresponding image areas. For example, in an image containing a worker wearing a helmet and a safety belt, the computer system identifies the location of the helmet and the safety belt through the target detection algorithm, and then randomly selects the helmet area and modifies the pixel value of the area (such as setting it to 0 or blurring it), so that the neural network needs to learn from other parts of the image and the remaining monitoring target features, enhancing its ability to understand and process image features in different situations. Take the feature shielding of a specific third-operation sensor monitoring image as an example. The image shows a scene on an aerial work platform with an operator wearing a helmet and a safety belt, and some work tools placed next to him. The computer system uses the method of feature shielding operations on multiple image blocks to divide the image into 16 image blocks of size 32×32, selects 4 of the image blocks by random numbers, and sets all pixel values ​​of these 4 image blocks to 0. After such processing, the image becomes a sample in the sample set of the operation sensor monitoring image. By performing similar feature masking operations on a large number of third-party operation sensor monitoring images, the computer system obtains a rich and diverse set of operation sensor monitoring image samples. These sample sets contain images with various feature masking conditions, providing sufficient data support for subsequent pre-training.

[0208] In step S3, the computer system performs pre-debugging (pre-training) on ​​the first branch network component based on the work sensor monitoring image sample set and the third work sensor monitoring image. The purpose of pre-training is to allow the first branch network component to preliminarily learn some common features and patterns in the image before the formal joint learning adjustment, reduce the time and difficulty of subsequent training, and improve the training efficiency and performance of the entire high-altitude work status monitoring neural network.

[0209] During the pre-training process, the computer system takes the sample set of the operation sensor monitoring image and the third operation sensor monitoring image as input and inputs them into the first branch network component. The first branch network component will perform data feature representation on these images, that is, through a series of neural network layers such as convolutional layers, pooling layers, and fully connected layers, the image data is converted into an abstract feature representation. In this process, the computer system adjusts the network parameters of the first branch network component through the back propagation algorithm according to the difference between the output result of the network and the expected output.

[0210] Assume that the first branch network component is a convolutional neural network (CNN) containing multiple convolutional layers C i , Pooling layer P j and the fully connected layer F k For the input image I, it is first processed by the convolution layer C1, which performs convolution operation with the image through the convolution kernel to extract local features in the image. Assume that the size of the convolution kernel is s×s and the step size is t.

[0211] Next, the pooling layer P j The output of the convolutional layer is downsampled to reduce the amount of data and computation while retaining important feature information. After being processed by the convolutional layer and the pooling layer, the data enters the fully connected layer F k The fully connected layer integrates the output features of the previous layer to obtain the final feature representation.

[0212] During the pre-training process, the computer system inputs the sample set of the operation sensor monitoring image and the third operation sensor monitoring image into the first branch network component to obtain the output result of the network. Then, a loss function is defined to measure the difference between the network output and the expected output. Commonly used loss functions include the mean square error (MSE) loss function.

[0213] For a simple first-branch network component, during the pre-training process, a sample of a work sensor monitoring image that has undergone a feature masking operation is input, and the network outputs a feature representation vector y. Given the expected output vector y* corresponding to the image, the computer system calculates the loss value as 0.5 through the MSE loss function. Then, the gradient is calculated through the back-propagation algorithm, and the weights and biases of the network are adjusted according to the gradient. After multiple such iterative trainings, the loss value of the network gradually decreases, such as to 0.1. This shows that the first-branch network component is constantly learning the features of the image and is gradually able to more accurately represent the data features of the input image.

[0214] By pre-training a large number of work sensor monitoring image sample sets and third work sensor monitoring images, the first branch network component can learn various feature patterns in the image, such as the appearance features of different safety equipment, the posture features of the operators, and the background features of the work environment. These pre-trained feature representation capabilities will provide a good foundation for the subsequent joint learning and adjustment of the high-altitude work status monitoring neural network in collaboration with the second branch network component, so that the neural network can detect and identify the status faster and more accurately when facing actual high-altitude work images, thereby improving the performance and reliability of the entire monitoring system. At the same time, the pre-training process also helps to reduce the risk of overfitting of the network in subsequent training, improve the generalization ability of the network, and enable it to better adapt to different high-altitude work scenarios and image data.

[0215] As an implementation manner, the step S2, performing a feature masking operation on each of the third operation sensor monitoring images, includes:

[0216] Mode 1: performing a random feature masking operation on the third operation sensor monitoring image;

[0217] or;

[0218] Mode 2: performing a feature masking operation on a plurality of image blocks in the third operation sensor monitoring image;

[0219] or;

[0220] Mode 3: performing feature shielding operation on multiple monitoring targets in the third operation sensor monitoring image.

[0221] In method 1, the computer system performs a random feature masking operation on the third task sensor monitoring image. The random feature masking operation refers to randomly selecting some pixels at the pixel level of the image and changing or masking the features of these pixels, thereby generating new image samples with different feature distributions. The purpose of this operation is to increase the diversity of data so that the neural network can learn a wider range of image features during the training process and improve its generalization ability.

[0222] A common technical means for a computer system to implement random feature masking operations is to use a random number generator. First, for a third-job sensor monitoring image, the computer system obtains its pixel matrix. Assuming that the image is M×N in size, each element I in its pixel matrix I ij Represents the color value of the pixel in the i-th row and j-th column of the image (for grayscale images, I_{ij} is a scalar value; for color images, I ij is a vector containing the values ​​of the red, green, blue, etc. color channels).

[0223] The computer system generates a set of random integers in the range of 1 to M×N through a random number generator. These random integers correspond to the pixel positions in the image. For example, suppose the generated random integers are k1, k2,…, ks, where s is the number of random numbers generated, and s can be set according to actual needs, generally determined by the image size and masking ratio. Then, the computer system modifies the pixel values ​​corresponding to these random positions.

[0224] One way to modify this is to set these pixel values ​​to 0. For example, for a 512×512 grayscale image of an aerial work scene, the computer system generates 100 random numbers, and the pixel positions corresponding to these random numbers are randomly distributed in the image. The computer system sets all the pixel values ​​at these positions to 0, thus completing a random feature masking operation and generating a new image sample. This new sample is based on the original image, and some pixel features are masked, so that the neural network needs to learn from the remaining pixel features during training, thereby enhancing its adaptability to different feature combinations.

[0225] Another way to mask random features is to perform a random transformation on randomly selected pixel values. For example, for a color image, a computer system can add a random offset to each color channel value of a randomly selected pixel. Assume that the pixel , randomly generate three offsets , then the modified pixel value is In this way, not only the features of the original pixels are masked, but also new random features are introduced, further enriching the diversity of the data.

[0226] In method 2, the computer system performs feature masking operations on multiple image blocks in the third operation sensor monitoring image. Image blocks are sub-regions of a certain size into which an image is divided. Feature masking of multiple image blocks can change the feature structure of the image at the local region level, prompting the neural network to learn the relationship between different local regions in the image and the association between overall features and local features.

[0227] The computer system will first determine how to divide the image into blocks. A common method is to divide the image into square or rectangular blocks of equal size. For example, for an M×N image, the computer system can divide it into blocks of size a×b, where a and b are positive integers, M is an integer multiple of a, and N is an integer multiple of b. The number of image blocks after division is .

[0228] Then, the computer system randomly selects multiple image blocks from these image blocks for feature masking. Assuming that a total of m image blocks are divided, the computer system generates a set of random integers l1, l2, …, l in the range of 1 to m through a random number generator.t , where t is the number of selected image blocks, and these random integers correspond to the image block indices to be masked.

[0229] For the selected image blocks, the computer system can use a variety of methods to perform feature masking. A simple method is to set all pixel values ​​of these image blocks to a fixed value, such as 0 or 255 (for grayscale images), or to a specific color vector (for color images). For example, a color third-job sensor monitoring image is divided into 16×16 image blocks, and a total of The computer system randomly selects 5 image blocks and sets all pixel values ​​of these 5 image blocks to [0, 0, 0], that is, black. In this way, in the generated new image samples, the regional features of these 5 black image blocks are completely blocked. The neural network needs to learn information from other unblocked image blocks, which helps it learn the feature distribution and mutual relationship of different regions in the image.

[0230] Another way is to blur the selected image block. The computer system can use the Gaussian blur algorithm to process the image block. The Gaussian blur algorithm achieves the blur effect by convolving a Gaussian kernel with the image block. For an a×b image block B, the Gaussian kernel G is a k×k matrix, where k is an odd number, and the central element is the largest and gradually decreases toward the surroundings. The calculation formula for the Gaussian kernel element is:

[0231] ;in is the standard deviation of the Gaussian distribution, which controls the degree of blur. After Gaussian blurring, the pixel values ​​of the image block change, and its features become blurred, thus achieving the purpose of feature masking. For example, applying the standard deviation to the selected image block = 2, the convolution operation is performed on the Gaussian kernel to blur the detailed features of these image blocks. The generated new image samples contain some blurred areas, which is a new feature combination for the neural network and helps to improve its adaptability to different feature situations.

[0232] In method 3, the computer system performs feature shielding operations on multiple monitoring targets in the third operation sensor monitoring image. Monitoring targets refer to objects related to operation safety in high-altitude operation scenes, such as safety helmets, safety belts, and operation tools. Feature shielding of monitoring targets can simulate the situation in actual scenes where the monitoring targets are partially invisible or the features are unclear, and enhance the neural network's ability to process incomplete information.

[0233] The computer system first needs to identify the monitoring targets in the image. This can be achieved through target detection algorithms, such as deep learning-based target detection algorithms such as Faster R-CNN and YOLO. These algorithms can classify and locate objects in the image and output the category and location information of each monitoring target in the image.

[0234] Assume that through the target detection algorithm, the computer system identifies three monitoring targets in a third operation sensor monitoring image: a safety helmet, a safety belt and a working tool, which are respectively , and get their position coordinates in the image ,in are the upper left and lower right corner coordinates of the monitoring target Ti respectively.

[0235] Then, the computer system randomly selects multiple monitoring targets for feature masking. Assume that the safety helmet T1 and the working tool T3 are randomly selected. For the selected monitoring targets, the computer system can use a variety of feature masking methods. One method is to set the pixel value of the image area corresponding to the monitoring target to 0. For example, for the image area where the safety helmet T1 is located , the computer system converts all pixel values ​​in this area into (in ) is set to 0, that is, the helmet area is turned black. Similarly, a similar operation is performed on the image area where the work tool T3 is located. In the new image samples generated in this way, the features of the helmet and the work tool are shielded, and the neural network needs to learn from the features of other parts of the image and the remaining monitoring targets (such as seat belts) to improve its ability to process incomplete target information.

[0236] Another feature shielding method is to occlude the monitoring target. The computer system can superimpose an occluding object, such as a rectangular or circular black area, on the image area of ​​the monitoring target. For example, on the image area of ​​the helmet T1, the computer system draws a black circle with a radius of r, covering part of the helmet area. In this way, the situation where the monitoring target is partially occluded in the actual scene is simulated, so that the neural network can learn how to judge and identify when the monitoring target features are incomplete.

[0237] The computer system performs feature masking operations on the third operation sensor monitoring image through methods 1, 2, and 3, and introduces different degrees of feature changes and masking at the pixel level, image block level, and monitoring target level, respectively, to generate a rich and diverse set of operation sensor monitoring image samples. These sample sets can provide diverse data for the pre-debugging of the first branch network component, so that the network can learn a wider range of image features and patterns during the pre-training process, improve the network's generalization ability and adaptability to different scenarios, and thus lay a solid foundation for the subsequent joint learning and adjustment of the high-altitude operation status monitoring neural network.

[0238] Figure 2 A hardware entity diagram of a computer system provided by an embodiment of the present invention is as follows Figure 2 As shown, the hardware entity of the computer system 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.

[0239] The above description is only an implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for monitoring the status of aerial work based on sensor data analysis, characterized in that: Used to perform status detection on the operation sensor monitoring image using a high-altitude operation status monitoring neural network, wherein the high-altitude operation status monitoring neural network includes a first branch network component and a second branch network component; The state detection method comprises: Acquiring a first operation sensor monitoring image; Loading the first operation sensor monitoring image into the first branch network component for data feature representation, and obtaining a first monitoring implicit representation corresponding to the first operation sensor monitoring image; The first monitoring implicit representation is loaded into the second branch network component, and the monitoring target state recognition is performed on the first operation sensor monitoring image to obtain a monitoring target state recognition result, wherein the monitoring target state recognition result is used to indicate the state relationship of the monitoring target in the first operation sensor monitoring image; The high-altitude operation status monitoring neural network is calibrated by the following steps: Acquire a second operation sensor monitoring image, a priori monitoring target state corresponding to the monitoring target in the second operation sensor monitoring image, and a priori involved image recognition result of the second operation sensor monitoring image; wherein the priori monitoring target state is used to indicate the state relationship of the monitoring target in the second operation sensor monitoring image, and the priori involved image recognition result is used to indicate the image recognition result of involved image recognition of the second operation sensor monitoring image; Loading the second operation sensor monitoring image into the first branch network component for data feature representation, and obtaining a third monitoring implicit representation corresponding to the second operation sensor monitoring image; The third monitoring implicit representation is loaded into the second branch network component, and the monitoring target state recognition is performed on the second operation sensor monitoring image to obtain the monitoring target state recognition result corresponding to the monitoring target in the second operation sensor monitoring image; The third monitoring implicit representation is loaded into the involved image recognition neural network, and the involved image recognition processing is performed on the second operation sensor monitoring image to obtain the involved image recognition result corresponding to the second operation sensor monitoring image; Based on the monitoring target state recognition result and the prior monitoring target state, and the involved image recognition result and the prior involved image recognition result, obtaining a calibration cost value of the calibration; Based on the adjustment cost value, network parameters of the first branch network component and the second branch network component are adjusted.

2. The method for monitoring the state of aerial work based on sensor data analysis according to claim 1 is characterized in that: Based on the first monitoring implicit representation, the first operation sensor monitoring image is subjected to monitoring target state recognition to obtain the monitoring target state recognition result, including: Performing feature comparison on the first operation sensor monitoring image in a preset feature library to obtain a second monitoring implicit representation corresponding to the first operation sensor monitoring image; performing implicit representation integration on the first monitoring implicit representation and the second monitoring implicit representation to obtain a fourth monitoring implicit representation; The fourth monitoring implicit representation is loaded into the second branch network component, and the monitoring target state recognition is performed on the first operation sensor monitoring image to obtain the monitoring target state recognition result.

3. The method for monitoring the state of aerial work based on sensor data analysis according to claim 2 is characterized in that: The first monitoring implicit representation includes a pixel implicit representation, and the second monitoring implicit representation includes a first image block implicit representation; The first monitoring implicit representation and the second monitoring implicit representation are implicitly integrated to obtain a fourth monitoring implicit representation, including: Performing image segmentation on the first operation sensor monitoring image; Based on the result obtained by the image segmentation, the implicit representation of the pixel is integrated to obtain an implicit representation of a second image block corresponding to the monitoring target in the first operation sensor monitoring image; The first image block implicit representation and the second image block implicit representation are implicitly integrated to obtain a fourth monitoring implicit representation.

4. The method for monitoring the state of aerial work based on sensor data analysis according to claim 1 is characterized in that: The involved image recognition neural network includes a monitoring target classification network; the loading of the third monitoring implicit representation into the involved image recognition neural network, performing involved image recognition processing on the second operation sensor monitoring image, and obtaining an involved image recognition result corresponding to the second operation sensor monitoring image, including: The third monitoring implicit representation is loaded into the monitoring target classification network, and the target classification result of the second operation sensor monitoring image is inferred to obtain the target classification inference result corresponding to the second operation sensor monitoring image.

5. The method for monitoring the state of aerial work based on sensor data analysis according to claim 1 is characterized in that: The involved image recognition neural network includes a monitoring target commonality measurement network; the loading of the third monitoring implicit representation into the involved image recognition neural network, performing involved image recognition processing on the second operation sensor monitoring image, and obtaining an involved image recognition result corresponding to the second operation sensor monitoring image, including: Arbitrarily segmenting the second operation sensor monitoring image to obtain a first target operation sensor monitoring image and a second target operation sensor monitoring image; Determining in the third monitoring implicit representation a first target implicit representation obtained based on the first target operation sensor monitoring image processing, and a second target implicit representation obtained based on the second target operation sensor monitoring image processing; The first target implicit representation and the second target implicit representation are loaded into the monitoring target commonality measurement network, and the image commonality degree of the first target operation sensor monitoring image and the second target operation sensor monitoring image is measured to obtain the image commonality measurement result of the first target operation sensor monitoring image and the second target operation sensor monitoring image.

6. The method for monitoring the state of aerial work based on sensor data analysis according to claim 1 is characterized in that: The step of obtaining a calibration cost value based on the monitoring target state recognition result and the prior monitoring target state, and the involved image recognition result and the prior involved image recognition result includes: Determining a first sub-adjustment cost value for performing monitoring target state recognition on the second operation sensor monitoring image based on the monitoring target state recognition result and the prior monitoring target state; Determining a second sub-adjustment cost value for performing an involved image recognition process on the second operation sensor monitoring image based on the involved image recognition result and the prior involved image recognition result; The first sub-adjustment cost value and the second sub-adjustment cost value are merged to obtain an adjusted adjustment cost value.

7. The method for monitoring the state of aerial work based on sensor data analysis according to claim 1, characterized in that: The method further comprises: Determining whether each of the second operation sensor monitoring images omits a corresponding a priori monitoring target state or a priori involved image recognition result; If the second operation sensor monitoring image omits the corresponding prior monitoring target state, skip the step of loading the third monitoring implicit representation into the second branch network component and performing monitoring target state identification on the second operation sensor monitoring image; or, if the second operation sensor monitoring image omits the corresponding prior involved image recognition result, skip the step of loading the third monitoring implicit representation into the involved image recognition neural network and performing involved image recognition processing on the second operation sensor monitoring image.

8. The method for monitoring the state of aerial work based on sensor data analysis according to claim 1, characterized in that: The method further comprises: Acquiring a plurality of third operation sensor monitoring images; Performing a feature masking operation on each of the third operation sensor monitoring images to obtain an operation sensor monitoring image sample set; Based on the operation sensor monitoring image sample set and the third operation sensor monitoring image, the first branch network component is pre-debugging.

9. The method for monitoring the state of aerial work based on sensor data analysis according to claim 8, characterized in that: The performing of a feature shielding operation on each of the third operation sensor monitoring images comprises: performing a random feature masking operation on the third operation sensor monitoring image; or; performing a feature masking operation on a plurality of image blocks in the third operation sensor monitoring image; or; A feature shielding operation is performed on the plurality of monitoring targets in the third operation sensor monitoring image.

10. A computer system comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 9 are implemented.

Images