Underwater image lightweight processing method and device

Abstract

The invention relates to the field of underwater image processing, and discloses an underwater image lightweight processing method and device, and the key points of the technical scheme are that the method comprises the following steps: selecting a teacher model and a student model, obtaining historical image data, dividing a data set, and training the selected teacher model by using the data set to obtain a solidified teacher model; respectively inputting the data set into a teacher model and a selected student model, and respectively outputting to obtain feature maps; screening the student model output feature map, and outputting a screened mask feature map; normalizing the mask feature map and an output feature map of the teacher model, and combining the normalized mask feature map and the output feature map to obtain a loss # imgabs0 #; decoupling the student model output feature graph and the teacher model feature graph, and solving a prediction loss # imgabs1 #; training a student model by using the data set, and calculating loss # imgabs2; combining and adding the weights of the # imgabs3 #, the # imgabs4 # and the # imgabs5 # to obtain a final loss # imgabs6 #, and carrying out round iteration to carry out distillation learning to obtain a finished product student model; and deploying and applying the finished student model.

Description

Technical Field

[0001] The present invention relates to the field of underwater image processing, and more specifically, to a method and device for lightweight underwater image processing. Background Art

[0002] Marine ranches use machine vision technology to observe the underwater environment of marine ranches, which has the advantages of long time, non-contact, and low cost. However, due to the limitations of underwater image quality and the influence of twisting, deformation, occlusion, and overlap of underwater creatures during movement, high-precision deep learning target detection models suitable for this environment are often more complex in structure, with more parameters in the model and large amount of calculation, making it difficult to run in real time on low-computing power devices.

[0003] At present, the B / S architecture is commonly used to solve these problems. That is, the front-end camera transmits the image back to the back-end server. The back-end server receives the data sent back by the front-end device and processes, identifies and analyzes the image by calling the built deep learning model recognition environment. This form requires a large amount of data to be transmitted to the cloud for calculation, which will result in high network overhead, high latency and the risk of data transmission being easily intercepted. As the number of front-end devices increases, the bandwidth required by operators will also gradually increase, and the deployment cost will increase exponentially.

[0004] In order to solve the problems of high data center load and high operator bandwidth occupancy in existing marine ranches, it is necessary to improve the system data processing process and improve efficiency. The entire back-end server data center is transferred to the front-end underwater equipment for decentralized deployment. However, this brings another problem, that is, the underwater recognition model inevitably becomes more complex in the process of achieving good performance, and the existing low-computing power edge devices are difficult to deploy and operate. Therefore, it is necessary to propose a lightweight processing algorithm for the model, realize the lightweight slimming of the model while ensuring the accuracy as much as possible, and deploy the obtained slimmed-down model in the edge device to accelerate the real-time operation. Summary of the invention

[0005] The purpose of the present invention is to provide a method and device for lightweight processing of underwater images, which can reduce energy consumption and improve operating efficiency while ensuring accuracy.

[0006] The above technical objectives of the present invention are achieved through the following technical solutions: A method for lightweight processing of underwater images, comprising the following steps: S1. Select a teacher model and a student model, obtain historical image data, divide the data set, and use the data set to train the selected teacher model to obtain a solidified teacher model; S2. Input the data set into the teacher model and the selected student model respectively, and output the feature maps respectively; filter the feature map output by the student model using the first set threshold, and output the filtered mask feature map; S3. Normalize the mask feature map output by the student model and the feature map output by the teacher model and combine them to calculate the loss ; S4. Decouple the feature map output by the student model from the feature map of the teacher model and calculate the prediction loss ; S5. Use the dataset to train the student model and calculate the loss ; S6. , , The final loss is obtained by combining and adding the respective weights , according to the final loss , distillation learning is performed in rounds of iterations, and the finished student model is output; S7. Deploy the finished student model to the edge device software for application.

[0007] As a preferred technical solution of the present invention, after acquiring the historical image data, the historical image data is preprocessed to obtain a standard image, and the preprocessing includes: image size conversion, image flipping, and image normalization.

[0008] As a preferred technical solution of the present invention, after obtaining the standard picture, each standard picture data is labeled according to the underwater animal species, and several underwater animal data sets are obtained. A preset proportion of data is selected from each underwater animal data set to set the training set and test set of the data set.

[0009] As a preferred technical solution of the present invention, the teacher model is a deep learning model whose network complexity and accuracy exceed a preset model selection threshold, and the student model is a deep learning model whose running speed and accuracy exceed a preset threshold.

[0010] As a preferred technical solution of the present invention, the feature map output by the student model is screened using a first set threshold, and the process of outputting the screened mask feature map is as follows: Assume that the feature map that exceeds the preset threshold is 1, and that that does not exceed is 0. and Represents the teacher model and the student model feature map; on the feature map, set a random mask to cover the feature, then the The pixel mask of the student feature map can be expressed as: ;in, is a random number in the feature map (0,1), , are the horizontal and vertical coordinates in the feature map, respectively. is a hyperparameter representing the masking ratio; Use the corresponding mask to cover the feature map of the student model, and fit the teacher feature map by generating pixels. The process expression is: , ; in, It means that the mask covers the student model feature map and aligns it with the teacher model feature map. represents the mask image, represents the student model feature map, Indicates that the aligned mask covers the feature map of the student model, Represents two convolutions and and an activation layer The protective layer Represents the mask feature map after multiple convolution and activation layers.

[0011] As a preferred technical solution of the present invention, the loss The process is: Normalize the mask data output by the student model and the feature map output by the teacher model; Then the loss L1 is obtained through the mean square error function, and its expression is: , is the sum of the total number of distillation layers, Indicates the number of channels, Represents the height of the feature map, represents the width of the feature map, For the distillation layer layer, For the channels, Indicates that the normalization process is completed. and Represent the feature maps of the student model and the teacher model respectively; loss The process of obtaining is: prediction loss It includes target class loss TCKD and non-target class loss NCKD, and the loss is calculated after prediction. , including: taking KL divergence as the loss function and decoupling it into target class and non-target class; for the loss of target class and non-target class, let: For each element The probability of a class, is the number of classes, For the The logit value of each class, For the The logit value of each class is passed through softmax, then: ; Let the target class binary probability be , the binary probability of the non-target class is , target class The logit value of , For non-target The logit value of each class is: , , set up is the independent model probability between non-target classes, and each element is calculated as: ; Among them, the prediction simulation is decoupled into target class and non-target class, and the and non-target classes Collectively referred to as , and Represents the teacher model and the student model, and brings them into the KL divergence for solution, then: ;in, Model for teachers The probability of an element, is the independent model probability between non-target classes of the teacher model, is the probability of independent models between non-target classes in the student model. After substituting it into the KL divergence, it is: ; Decouple TCKD and NCKD again and add hyperparameters and , we get the formula for predicting simulation loss: ; Final loss The expression is: ,in, , , are their respective pre-set weights.

[0012] As a preferred technical solution of the present invention, the deployment process of S7 is: installing the front-end environment in the edge device; Convert the final student model into a pt file, then into a TorchScript file, and finally into an engine file, and perform hardware compilation on the obtained engine file; Embed the engine file into the edge device software; The edge device software receives the message commands sent by the water display and control station to activate the corresponding functions, and the student model is used to process and analyze the data to obtain the classification information and number of underwater organisms.

[0013] A lightweight underwater image processing device comprises: a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor implements the above method when executing the computer program.

[0014] In summary, the present invention has the following beneficial effects: it realizes lightweight slimming of the model while ensuring accuracy as much as possible. The slimmed-down model is deployed in the edge device to accelerate the real-time operation, which can save the time of building a high-performance computing data center in the cloud. In comparison, it is only necessary to build a computer storage array and a computer with software-controlled display as before. In terms of energy consumption, the energy consumption required by the remote edge device is more energy-efficient than that of the high-performance computing data center. The present invention also deploys the data processing part in the remote edge device. When the number of ocean ranch sites gradually increases, it will not be limited by the performance of the cloud computing center, which is more conducive to improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a flow chart of deployment of the finished student model of the present invention. DETAILED DESCRIPTION

[0016] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0017] The present invention provides a method for lightweight processing of underwater images. Figure 1As shown in the figure, the processed underwater image is substituted into the backbone network of the teacher model and the student model, and the feature maps of the teacher model and the student model are converted into the feature maps of the teacher model and the student model and enter their respective projection layers. In the projection layer, the feature map is copied, and the feature map of the student model is sent to the generated feature block after being processed to generate mask features, and then the normalized mask features are formed through the normalization module. After the teacher model feature map is copied, it is passed through the normalization module to form the normalized teacher feature. The normalized mask feature and the normalized teacher feature are distilled to obtain the feature loss. The copied teacher and student model feature maps that have not been processed in the projection layer are processed in the classification head. The classification head predictions of the teacher and the student obtained after processing are copied and divided into target classes and non-target classes through the decoupling module. The target classes and non-target classes of the teacher and student models are distilled one by one to obtain the prediction loss. The feature map copied by the student model that has not been processed by the projection layer and the classification head is input into the regression head, and combined with the predictions copied in the classification head of the student model to obtain the original loss. The feature loss, prediction loss and original loss are combined with their respective set weights to obtain the final total loss.

[0018] The specific steps include: S1. Select the teacher model and the student model. The teacher model is a deep learning model whose network complexity and accuracy exceed the preset model selection threshold. The student model is a deep learning model whose running speed and accuracy exceed the preset threshold. Compared with the student model, the teacher model has a more complex model structure and higher accuracy. Compared with the teacher model, the student model has a simpler model network, lower accuracy, but faster speed. It comes from the existing model library or is built according to the current scenario.

[0019] The teacher model and the student model use accuracy as an indicator. The accuracy indicators in target detection are mAP, APS, APM, and APL. The main one is mAP, which is the average precision (mAP) measurement on the IoU threshold from 0.5 to 0.95 in all classes. APS, APM, and APL are the accuracy of small, medium, and large areas. When choosing, you can relatively accurately select the teacher model and student model that meet the accuracy requirements of the current scene.

[0020] Acquire historical image data and divide the data set, including: preprocess the historical image data to obtain standard images, the preprocessing includes: image size conversion, image flipping, and image normalization; After obtaining the standard images, each standard image data is labeled according to the underwater animal species, and several underwater animal data sets are obtained. A preset proportion of data is selected from each underwater animal data set to set the training set and test set of the data set. The species are labeled according to the target animal species, including Otaki six-line fish, black sea bream, black carp, starfish, spindle amberjack, beautiful butterfly fish, eight-band butterfly fish, vertical band cardinalfish, etc.

[0021] The selected teacher model is trained using the data set to obtain a solidified teacher model.

[0022] S2. Input the data set into the teacher model and the selected student model respectively, and output the feature maps respectively; filter the feature map output by the student model using the first set threshold, and output the filtered mask feature map; The feature map output by the student model is filtered using the first set threshold, and the process of outputting the filtered mask feature map is: Assume that the feature map that exceeds the preset threshold is 1, and that that does not exceed is 0. and Represent the lth feature map of the teacher model and the student model respectively; on the feature map, a random mask is set to cover the feature, then the lth student feature map pixel mask can be expressed as: ;in, is a random number in the feature map (0,1), , are the horizontal and vertical coordinates in the feature map, respectively. is a hyperparameter representing the masking ratio; Use the corresponding mask to cover the feature map of the student model, and fit the teacher feature map by generating pixels. The process expression is: , ; in, It means that the mask covers the student model feature map and aligns it with the teacher model feature map. represents the mask image, represents the student model feature map, Indicates that the aligned mask covers the feature map of the student model, Represents two convolutions and and an activation layer The protective layer Represents the mask feature map after multiple convolution and activation layers.

[0023] S3. Normalize the mask feature map output by the student model and the feature map output by the teacher model and combine them to calculate the loss , i.e. feature loss; Seeking loss The process is: Normalize the mask data output by the student model and the feature map output by the teacher model; Then the loss L1 is obtained through the mean square error function, and its expression is: , is the sum of the total number of distillation layers, Indicates the number of channels, Represents the height of the feature map, represents the width of the feature map, For the distillation layer layer, For the channels, Indicates that the normalization process is completed. and Represent the feature maps of the student model and the teacher model respectively; S4. Decouple the feature map output by the student model from the feature map of the teacher model and calculate the loss , i.e. predicted loss; loss The process of obtaining is: prediction loss It includes target class loss TCKD and non-target class loss NCKD, and the loss is calculated after prediction. , including: taking KL divergence as the loss function and decoupling it into target class and non-target class; for the loss of target class and non-target class, let: For each element The probability of a class, is the number of classes, For the The logit value of each class, For the The logit value of each class is passed through softmax, then: ; Let the target class binary probability be , the binary probability of the non-target class is , target class The logit value of , For non-target The logit value of each class is: , , set up is the independent model probability between non-target classes, and each element is calculated as: ; Among them, the prediction simulation is decoupled into target class and non-target class, and the and non-target classes Collectively referred to as , and Represents the teacher model and the student model, and brings them into the KL divergence for solution, then: ;in, Model for teachers The probability of an element, is the independent model probability between non-target classes of the teacher model, is the probability of independent models between non-target classes in the student model. After substituting it into the KL divergence, it is: ; Decouple TCKD and NCKD again and add hyperparameters and , we get the formula for predicting simulation loss: ; S5. Use the dataset to train the student model and calculate the loss , i.e., the original loss; S6. , , The final loss is obtained by combining and adding the respective weights , according to the final loss , distillation learning is performed in rounds of iterations, and the finished student model is output; Final loss The expression is: ,in, , , are their respective pre-set weights.

[0024] The purpose of S1-6 is to use the teacher model and the data set to train the student model together. The student model is the final output. The main basis for selecting the student model is the actual running speed and accuracy. The model network is simple and shallow, and the corresponding speed is fast, but the accuracy will be low. Therefore, through the above steps, distillation learning can ensure the speed of the model while improving its accuracy as much as possible. Because in distillation learning, the framework of the teacher and student models is fixed.

[0025] In the actual student model distillation process, it is used to achieve the distillation of the target functional model, and various functional student models can be distilled, including underwater fish posture observation models and fish body length estimation models.

[0026] S7. Deploy the finished student model to the edge device software for application.

[0027] like Figure 2 As shown, the deployment process of S7 is as follows: installing the pre-environment in the edge device; Convert the final student model into a pt file, then into a TorchScript file, and finally into an engine file, and perform hardware compilation on the obtained engine file; Embed the engine file into the edge device software; The edge device software receives the message command sent by the water display and control machine to activate the corresponding function, and the student model is used to process and analyze the data to obtain the classification information and number of underwater organisms, which are then displayed on the water display and control machine.

[0028] In the actual application scenario of the present invention, the hardware includes a self-contained base platform, an underwater high-definition camera, an underwater fluorescent camera, a multi-parameter water quality meter, a submarine cable, an underwater LED light, an anti-biological attachment UV light, a portable coral distribution survey instrument, a tide meter, a CTD, a display and control camera and other devices.

[0029] The self-contained base platform is a pedestal placed at the bottom of the ocean ranch, on which various instruments are placed. The underwater data processing equipment is also placed on the platform. The underwater data processing equipment is connected to underwater high-definition cameras and other equipment and is responsible for receiving data transmitted by the equipment. The display and control position on the water is connected to the water data center through cables and connected to underwater data processing equipment and other devices.

[0030] Underwater high-definition cameras and underwater fluorescent cameras are deployed on self-contained base platforms to observe the number of individuals of underwater fish and economic animals within a certain range. The collected video images are transmitted to the underwater edge device for data processing, and the results obtained are transmitted back to the water data center and display control machine together with the results obtained by multi-parameter water quality meters, tide level meters and other equipment for real-time monitoring.

[0031] In S7, the student model output by the software is compiled and deployed faster on the hardware side, which can further increase the speed and reduce the equipment operating load. The cloud server data processing center is migrated to each front-end device, and the underwater data is processed and stored at the front end, and the recognition data is transmitted back at regular intervals. The remote data center device only has display control and storage functions, which can control remote functions and display remote recognition results in real time.

[0032] Corresponding to the above method, the present invention also provides an underwater image lightweight processing device, including: a processor and a memory, the memory stores a computer program executable by the processor, and the processor implements the methods S1-S7 when executing the computer program.

[0033] The lightweight processing method of underwater images of the present invention realizes the lightweight slimming of the model while ensuring the accuracy as much as possible. The obtained slimmed-down model is deployed in the edge device to accelerate the real-time operation, which can save the role of building a high-performance computing data center in the cloud. In comparison, it is only necessary to build a computer storage array and a computer with software-controlled display as before. In terms of energy consumption, the energy consumption required by the remote edge device is more energy-efficient than that of the high-performance computing data center. The present invention also deploys the data processing part in the remote edge device. When the number of marine ranch sites gradually increases, it will not be limited by the performance of the cloud computing center, which is more conducive to improving efficiency.

[0034] The advantages of the present invention are as follows: the present invention increases the model from running on a cloud server to running on an embedded edge device, which has the advantages of distributed load and high real-time performance, and effectively avoids the problem of high load on the water data center and high bandwidth usage of operators. In the case of a large number of underwater sites in the ocean ranch, the display and control software can also be guaranteed to run in real time, and will not freeze or flash back due to heavy load, thereby improving efficiency. The water-based shore-based data center is replaced with a distributed edge device processing unit with lower energy consumption, saving a certain amount of energy consumption.

[0035] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A lightweight processing method for underwater images, Its characteristics are: The steps include: S1. Select a teacher model and a student model, obtain historical image data, divide the data set, and use the data set to train the selected teacher model to obtain a solidified teacher model; S2. Input the data set into the teacher model and the selected student model respectively, and output the feature maps respectively; filter the feature map output by the student model using the first set threshold, and output the filtered mask feature map; S3. Normalize the mask feature map output by the student model and the feature map output by the teacher model and combine them to calculate the loss ; S4. Decouple the feature map output by the student model from the feature map of the teacher model and calculate the prediction loss ; S5. Use the dataset to train the student model and calculate the loss ; S6. , , The final loss is obtained by combining and adding the respective weights , according to the final loss , distillation learning is performed in rounds of iterations, and the finished student model is output; S7. Deploy the finished student model to the edge device software for application.

2. The method for lightweight processing of underwater images according to claim 1, Its characteristics are: After the historical image data is acquired, the historical image data is preprocessed to obtain a standard image. The preprocessing includes: image size conversion, image flipping, and image normalization.

3. The method for lightweight processing of underwater images according to claim 2, Its characteristics are: After obtaining the standard pictures, each standard picture data is labeled according to the underwater animal species, and several underwater animal data sets are obtained. A preset proportion of data is selected from each underwater animal data set to set the training set and test set of the data set.

4. The method for lightweight processing of underwater images according to claim 3, Its characteristics are: The teacher model is a deep learning model whose network complexity and accuracy exceed a preset model selection threshold, and the student model is a deep learning model whose running speed and accuracy exceed a preset threshold.

5. The method for lightweight processing of underwater images according to claim 4, Its characteristics are: The feature map output by the student model is filtered using the first set threshold, and the process of outputting the filtered mask feature map is: Assume that the feature map that exceeds the preset threshold is 1, and that that does not exceed is 0. and Represents the teacher model and the student model feature map; on the feature map, set a random mask to cover the feature, then the The pixel mask of the student feature map can be expressed as: ;in, is a random number in the feature map (0,1), , are the horizontal and vertical coordinates in the feature map, respectively. is a hyperparameter representing the masking ratio; Use the corresponding mask to cover the feature map of the student model, and fit the teacher feature map by generating pixels. The process expression is: ， ； in, It means that the mask covers the student model feature map and aligns it with the teacher model feature map. represents the mask image, represents the student model feature map, Indicates that the aligned mask covers the feature map of the student model, Represents two convolutions and and an activation layer The protective layer Represents the mask feature map after multiple convolution and activation layers.

6. The method for lightweight processing of underwater images according to claim 5, Its characteristics are: Seeking loss The process is: Normalize the mask data output by the student model and the feature map output by the teacher model; Then the loss L1 is obtained through the mean square error function, and its expression is: , is the sum of the total number of distillation layers, Indicates the number of channels, Represents the height of the feature map, represents the width of the feature map, For the distillation layer layer, For the channels, Indicates that the normalization process is completed. and Represent the feature maps of the student model and the teacher model respectively; loss The process of obtaining is: prediction loss It includes target class loss TCKD and non-target class loss NCKD, and the loss is calculated after prediction. , including: taking KL divergence as the loss function and decoupling it into target class and non-target class; for the loss of target class and non-target class, let: For each element The probability of a class, is the number of classes, For the The logit value of each class, For the The logit value of each class is passed through softmax, then: ； Let the target class binary probability be , the binary probability of the non-target class is , target class The logit value of , For non-target The logit value of each class is: ， ， set up is the independent model probability between non-target classes, and each element is calculated as: ； Among them, the prediction simulation is decoupled into target class and non-target class, and the and non-target classes Collectively referred to as , and Represents the teacher model and the student model, and brings them into the KL divergence for solution, then: ;in, Model for teachers The probability of an element, is the independent model probability between non-target classes of the teacher model, is the probability of independent models between non-target classes in the student model. After substituting it into the KL divergence, it is: ； Decouple TCKD and NCKD again and add hyperparameters and , we get the formula for predicting simulation loss: ； Final loss The expression is: ,in, , , are their respective pre-set weights.

7. The method for lightweight processing of underwater images according to claim 6, Its characteristics are: The deployment process of S7 is as follows: Install the front-end environment in the edge device; Convert the final student model into a pt file, then into a TorchScript file, and finally into an engine file, and perform hardware compilation on the obtained engine file; Embed the engine file into the edge device software; The edge device software receives the message commands sent by the water display and control station to activate the corresponding functions, and the student model is used to process and analyze the data to obtain the classification information and number of underwater organisms.

8. A lightweight underwater image processing device, Its characteristics are: include: A processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images