Image processing method and device based on deep learning, equipment and medium

Abstract

The application relates to a deep learning-based image processing method, device, equipment and medium. The method comprises the following steps: acquiring a to-be-processed dynamic video stream, wherein the to-be-processed dynamic video stream comprises to-be-processed images of each video frame; acquiring a hardware resource state parameter corresponding to a video frame of the to-be-processed image, and calculating a corresponding hardware state value of each to-be-processed image based on the hardware resource state parameter; calculating the image complexity and the interframe difference degree of the to-be-processed image; performing image processing deep learning model state judgment on the to-be-processed image based on the hardware state value, the image complexity and the interframe difference degree, and generating an image processing deep learning model state judgment result of the to-be-processed image. The method can realize hardware adaptation and dynamic model architecture fine-tuning, and significantly improve the model inference efficiency, image processing quality and hardware resource utilization rate of video stream processing.

Description

Technical Field

[0001] This application belongs to the field of image processing, and in particular relates to an image processing method, apparatus, device and medium based on deep learning. Background Technology

[0002] Deep learning image processing technology refers to a technical system based on deep neural network models that automatically extracts image features through multi-layer nonlinear transformations and completes processing tasks such as image classification, object detection, noise reduction, and enhancement. Its core advantage lies in its strong representation capability of complex image features. Dynamic video stream image processing refers to image processing technology that processes video data streams composed of consecutive frames frame by frame or in batches, requiring a balance between processing accuracy and real-time performance.

[0003] With the deep penetration of computer vision technology into fields such as intelligent monitoring, autonomous driving, and smart terminal interaction, the demand for dynamic video stream image processing is growing rapidly. Deep learning image processing technology, with its superior processing performance, has gradually replaced traditional image processing algorithms as the mainstream solution. Traditional dynamic video stream processing often adopts a fixed model architecture and static resource adaptation mode, lacking an adaptive mechanism to dynamic scene changes and failing to adapt in real time to the dynamic changes in video images and the real-time fluctuations in hardware resources. Furthermore, existing fixed model architectures struggle to meet the processing needs of images with varying complexity; complex images are prone to accuracy degradation due to insufficient model representation capabilities, while simple images suffer from hardware resource waste due to model redundancy. In addition, static hardware adaptation strategies cannot cope with hardware load fluctuations, severely restricting dynamic video stream image processing in scenarios with high real-time requirements. Summary of the Invention

[0004] Therefore, it is necessary to provide a deep learning-based image processing method, apparatus, device, and medium that can achieve hardware adaptation and dynamic model architecture fine-tuning to address the above-mentioned technical problems.

[0005] Firstly, this application provides a deep learning-based image processing method, including:

[0006] Acquire the dynamic video stream to be processed, which includes the images to be processed for each video frame;

[0007] Obtain the hardware resource status parameters corresponding to the video frames of the images to be processed, and calculate the corresponding hardware status values ​​of each image to be processed based on the hardware resource status parameters.

[0008] Calculate the image complexity and inter-frame difference of the image to be processed;

[0009] Based on hardware state values, image complexity, and inter-frame differences, the image processing deep learning model state determination is performed on the image to be processed, generating the image processing deep learning model state determination result for the image to be processed.

[0010] In one embodiment, the image processing deep learning model state determination result includes convolutional module structure determination result, channel pruning state determination result, and precision quantization determination result. The image processing deep learning model state determination is performed based on hardware state values, image complexity, and inter-frame difference, generating the image processing deep learning model state determination result, including:

[0011] Based on the hardware status values ​​and combined with the preset hardware status threshold range, the hardware status corresponding to the image to be processed is divided.

[0012] The selection range of the convolution module structure determination result and the selection range of the channel pruning state determination result are set based on the hardware status.

[0013] Based on image complexity and inter-frame difference, the final convolutional module structure determination result and the final channel pruning state determination result are selected from the selection range of convolutional module structure determination result and the selection range of channel pruning state determination result, respectively.

[0014] The final convolutional module structure determination results and the final channel pruning state determination results are summarized to generate the image processing deep learning model state determination results.

[0015] In one embodiment, the hardware states include low load, medium load, and high load states; the convolutional module structure determination results include the structure determination results of basic convolutional parallel depthwise separable convolutional modules, the structure determination results of basic convolutional modules, and the structure determination results of depthwise separable convolutional modules; the channel pruning state determination results include the determination results of reduced pruning degree, normal pruning degree, and increased pruning degree; the selection range of the convolutional module structure determination results includes the selection range of low load convolutional module structure determination results, the selection range of medium load convolutional module structure determination results, and the selection range of high load convolutional module structure determination results; and the selection range of the channel pruning state determination results includes the selection range of low load channel pruning state determination results, the selection range of medium load channel pruning state determination results, and the selection range of high load channel pruning state determination results.

[0016] The selection range for low-load convolutional module structure determination results includes the structure determination results of basic convolution parallel depthwise separable convolutional modules and the structure determination results of basic convolutional modules; the selection range for medium-load convolutional module structure determination results includes the structure determination results of basic convolutional modules and depthwise separable convolutional modules; and the selection range for high-load convolutional module structure determination results includes the structure determination results of depthwise separable convolutional modules.

[0017] The selection range for the pruning status determination results of the low-load channel includes the determination results of the reduced pruning degree and the normal pruning degree; the selection range for the pruning status determination results of the medium-load channel includes the determination results of the normal pruning degree and the determination results of the increased pruning degree; and the selection range for the pruning status determination results of the high-load channel includes the determination results of the increased pruning degree.

[0018] The selection range for the convolution module structure determination result and the selection range for the channel pruning state determination result are based on the hardware state settings, including:

[0019] If the hardware is in a low-load state, set the selection range of the convolution module structure determination result to the low-load convolution module structure determination result, and set the selection range of the channel pruning state determination result to the low-load channel pruning state determination result.

[0020] If the hardware state is medium load, set the selection range of the convolution module structure determination result to the medium load convolution module structure determination result, and set the selection range of the channel pruning state determination result to the medium load channel pruning state determination result.

[0021] If the hardware is under high load, set the selection range of the convolution module structure determination result to the high load convolution module structure determination result selection range, and set the selection range of the channel pruning state determination result to the high load channel pruning state determination result selection range.

[0022] In one embodiment, the deep learning-based image processing method further includes:

[0023] If the final convolutional module structure determination result is the basic convolutional parallel depth-separable convolutional module structure determination result, then set the convolutional module of the image processing deep learning model corresponding to the image to be processed as the basic convolutional parallel depth-separable convolutional module.

[0024] If the final convolutional module structure determination result is the basic convolutional module structure determination result, then the convolutional module of the image processing deep learning model corresponding to the image to be processed will be set as the basic convolutional module.

[0025] If the final convolutional module structure determination result is the depth-separable convolutional module structure determination result, then set the convolutional module of the image processing deep learning model corresponding to the image to be processed as a depth-separable convolutional module.

[0026] If the final channel pruning state determination result is a reduced pruning degree state determination result, the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed will be set to the reduced pruning degree channel pruning strategy.

[0027] If the final channel pruning state determination result is the normal pruning degree state determination result, the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed will be set to the normal pruning degree channel pruning strategy.

[0028] If the final channel pruning state determination result is an increased pruning degree state determination result, the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed will be set to the increased pruning degree channel pruning strategy.

[0029] In one embodiment, based on image complexity and inter-frame difference, the convolutional module structure determination result and the channel pruning state determination result are selected from the selection range of the convolutional module structure determination result and the selection range of the channel pruning state determination result, respectively, including:

[0030] Based on image complexity and inter-frame difference, and combined with the convolution module structure determination weight and pruning degree determination weight, the required values ​​of convolution module structure complexity and pruning degree model complexity are calculated respectively. Among them, the convolution module structure determination weight for image complexity is greater than the convolution module structure determination weight for inter-frame difference, and the pruning degree determination weight for image complexity is less than the pruning degree determination weight for inter-frame difference.

[0031] The convolutional module structure determination result with the smallest difference between the actual value of the convolutional module structure complexity and the required value of the convolutional module structure complexity within the range of convolutional module structure determination results is selected as the final convolutional module structure determination result.

[0032] The channel pruning state determination result with the smallest difference between the actual value of the pruning degree model complexity and the required value of the pruning degree model complexity within the selected range of channel pruning state determination results is selected as the final channel pruning state determination result.

[0033] In one embodiment, the hardware resource status parameters include computing resource utilization and memory bandwidth utilization, and the hardware status value is calculated by weighted summation of computing resource utilization and memory bandwidth utilization.

[0034] In one embodiment, the expressions for image complexity and inter-frame variability are:

[0035]

[0036]

[0037] In the formula, and The video frame number is determined by the image complexity and the video frame number, respectively. Inter-frame differences and These are the pixel height and pixel width of the image to be processed, respectively. and These are the texture complexity weights and gradient complexity weights, respectively. For the image to be processed at the pixel point Local binary pattern value at that location, and The image to be processed is located at the pixel level. The horizontal and vertical gradient values ​​are obtained by the Sobel operator. This represents the total number of video frames within the time window for inter-frame difference analysis. The video frame number is The image to be processed at the pixel point Pixel value at that location, The video frame number is The image to be processed at the pixel point The pixel value at that location.

[0038] Secondly, this application also provides a deep learning-based image processing apparatus, comprising:

[0039] The image acquisition module is used to acquire the dynamic video stream to be processed, which includes the images to be processed for each video frame.

[0040] The hardware status monitoring module is used to obtain the hardware resource status parameters corresponding to the video frames of the images to be processed, and to calculate the corresponding hardware status value of each image to be processed based on the hardware resource status parameters.

[0041] The image parameter calculation module is used to calculate the image complexity and inter-frame difference of the image to be processed.

[0042] The model parameter adjustment module is used to determine the state of the deep learning model for image processing based on hardware state values, image complexity, and inter-frame differences, and to generate the state determination result of the deep learning model for image processing of the image to be processed.

[0043] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method as described in any of the first aspects of this application.

[0044] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the first aspects of this application.

[0045] The aforementioned deep learning-based image processing methods, devices, equipment, and media, by real-time acquisition of hardware resource status parameters of the video frames corresponding to the image to be processed and calculation of hardware status values, can capture the dynamic operating status of the hardware, accurately quantify the real-time hardware load, and ensure efficient scheduling and stable operation of hardware resources during image processing. By calculating the image complexity and inter-frame difference of the image to be processed, it can deeply mine the feature attributes of single-frame images and the changing patterns of consecutive frame images, quantify image processing requirements, improve the targeting of image processing strategies, increase hardware resource utilization, enhance the dynamic allocation capability of hardware resources, and improve the matching degree between image processing strategies and actual image requirements. By fusing hardware status values, image complexity, and inter-frame difference for image processing deep learning model status determination, it can dynamically and collaboratively adapt hardware operating capabilities and image processing requirements, thereby balancing the accuracy, real-time requirements, and hardware device status of dynamic video stream processing, significantly improving the model inference efficiency, image processing quality, and hardware resource utilization of video stream processing. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart illustrating a deep learning-based image processing method provided in one embodiment of this application. Figure 1 ;

[0048] Figure 2 A flowchart illustrating a deep learning-based image processing method provided in one embodiment of this application. Figure 2 ;

[0049] Figure 3 This is a schematic diagram of the structure of an image processing device based on deep learning, provided in one embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] In one exemplary embodiment of this application, such as Figure 1As shown, a deep learning-based image processing method is provided. This embodiment illustrates the application of this method to an image processing terminal. It is understood that this method can also be applied to an image processing server, and further to an image processing system including both an image processing terminal and an image processing server, implemented through the interaction between the two. In this embodiment, the method includes the following steps:

[0052] Step S101: Obtain the dynamic video stream to be processed, which includes the images to be processed for each video frame.

[0053] Specifically, the image processing terminal can acquire a continuous dynamic video stream to be processed through video acquisition devices deployed in the target scene. The image processing terminal can perform frame-level splitting on the received dynamic video stream and extract the image to be processed from each independent video frame according to the time sequence of the video stream.

[0054] Optionally, the image processing terminal can record the video frame number and acquisition timestamp corresponding to each image to be processed.

[0055] To illustrate, in intelligent monitoring scenarios, the image processing terminal can establish a communication connection with video acquisition devices such as network cameras deployed in the monitoring area, and receive dynamic video streams to be processed in real time from the video acquisition devices through standard transmission protocols such as RTSP and HTTP; in autonomous driving scenarios, the image processing terminal can read dynamic video streams to be processed collected by vehicle-mounted vision sensors and / or vehicle-mounted remote sensing sensors; in intelligent terminal interaction scenarios, the image processing terminal can receive dynamic video streams to be processed transmitted from mobile devices through USB interfaces, serial communication interfaces, wireless projection, and wireless communication.

[0056] Step S102: Obtain the hardware resource status parameters corresponding to the video frames of the images to be processed, and calculate the corresponding hardware status values ​​of each image to be processed based on the hardware resource status parameters.

[0057] Specifically, the image processing terminal can obtain the hardware resource status parameters corresponding to the video frames of the images to be processed in the current operating environment in real time through the hardware interface. The image processing terminal can perform weighted summation of the hardware resource status parameters corresponding to the video frames of the images to be processed to calculate the corresponding hardware status value of each image to be processed.

[0058] Optionally, hardware resource status parameters may include computing resource utilization and memory bandwidth utilization.

[0059] Step S103: Calculate the image complexity and inter-frame difference of the image to be processed.

[0060] Optionally, the image processing terminal can calculate the image complexity and inter-frame difference of the image to be processed based on a preset algorithm.

[0061] Step S104: Based on hardware state values, image complexity, and inter-frame difference, perform image processing deep learning model state determination on the image to be processed, and generate the image processing deep learning model state determination result for the image to be processed.

[0062] For example, the image processing terminal can compare the calculated hardware state value corresponding to the image to be processed with a preset hardware state threshold range to classify the hardware state corresponding to the image to be processed. The image processing terminal can set the selection range for the convolutional module structure determination result and the channel pruning state determination result corresponding to the image to be processed based on the hardware state of the image to be processed. The image processing terminal can select the final convolutional module structure determination result and the final channel pruning state determination result from the selection range of the convolutional module structure determination result and the selection range of the channel pruning state determination result based on the image complexity and inter-frame difference. The image processing terminal can use the final convolutional module structure determination result and the final channel pruning state determination result as the state determination result of the deep learning model for image processing of the image to be processed.

[0063] Optionally, hardware states may include, but are not limited to, low load, medium load, or high load states.

[0064] Indicatively, the final convolutional module structure determination result can be used to adjust the structure of the convolutional modules of the image processing deep learning model corresponding to the image to be processed. The final channel pruning state determination result can be used to adjust the channel pruning fine-tuning strategy of the image processing deep learning model corresponding to the image to be processed.

[0065] In the aforementioned deep learning-based image processing method, by real-time acquisition of hardware resource status parameters of the video frames corresponding to the image to be processed and calculation of hardware status values, the dynamic operating status of the hardware can be captured, the real-time hardware load can be accurately quantified, and the efficient scheduling and stable operation of hardware resources during image processing can be ensured. By calculating the image complexity and inter-frame difference of the image to be processed, the feature attributes of a single frame image and the changing patterns of consecutive frames can be deeply explored, the image processing requirements can be quantified, the pertinence of the image processing strategy can be improved, the utilization rate of hardware resources can be increased, the dynamic allocation capability of hardware resources can be enhanced, and the matching degree between the image processing strategy and the actual image requirements can be improved. By fusing hardware status values, image complexity, and inter-frame difference for image processing deep learning model status determination, the hardware operating capabilities and image processing requirements can be dynamically and collaboratively adapted, thereby balancing the accuracy, real-time requirements, and hardware device status of dynamic video stream processing, significantly improving the model inference efficiency, image processing quality, and hardware resource utilization of video stream processing.

[0066] In an optional embodiment of this application, the state determination result of the image processing deep learning model may include the convolution module structure determination result, the channel pruning state determination result, and the accuracy quantization determination result. Please refer to [reference needed]. Figure 1 and Figure 2 Step S104 involves determining the state of the image processing deep learning model based on hardware state values, image complexity, and inter-frame differences, generating a state determination result for the image processing deep learning model. This result may include:

[0067] Step S204: Based on the hardware status value and combined with the preset hardware status threshold range, the hardware status corresponding to the image to be processed is divided.

[0068] Step S205: Based on the hardware status, set the selection range of the convolution module structure determination result and the selection range of the channel pruning status determination result.

[0069] Step S206: Based on the image complexity and inter-frame difference, the final convolutional module structure determination result and the final channel pruning state determination result are selected from the selection range of the convolutional module structure determination result and the selection range of the channel pruning state determination result, respectively.

[0070] Step S207: Summarize the final convolutional module structure determination results and the final channel pruning state determination results to generate the image processing deep learning model state determination results.

[0071] In an optional embodiment of this application, the hardware state may include a low-load state, a medium-load state, and a high-load state. The convolutional module structure determination result may include the basic convolutional parallel depthwise separable convolutional module structure determination result, the basic convolutional module structure determination result, and the depthwise separable convolutional module structure determination result. The channel pruning state determination result may include the reduced pruning degree state determination result, the normal pruning degree state determination result, and the increased pruning degree state determination result. The selection range of the convolutional module structure determination result may include the low-load convolutional module structure determination result selection range, the medium-load convolutional module structure determination result selection range, and the high-load convolutional module structure determination result selection range. The selection range of the channel pruning state determination result may include the low-load channel pruning state determination result selection range, the medium-load channel pruning state determination result selection range, and the high-load channel pruning state determination result selection range.

[0072] The selection range for the low-load convolutional module structure determination results can include the structure determination results of the basic convolutional parallel depthwise separable convolutional module and the structure determination results of the basic convolutional module. The selection range for the medium-load convolutional module structure determination results can include the structure determination results of the basic convolutional module and the structure determination results of the depthwise separable convolutional module. The selection range for the high-load convolutional module structure determination results can include the structure determination results of the depthwise separable convolutional module.

[0073] The selection range for the pruning status determination results of the low-load channel can include the determination results of reduced pruning degree and normal pruning degree; the selection range for the pruning status determination results of the medium-load channel can include the determination results of normal pruning degree and increased pruning degree; and the selection range for the pruning status determination results of the high-load channel can include the determination results of increased pruning degree.

[0074] The selection range for the convolution module structure determination result and the channel pruning state determination result can be set based on the hardware status, and may include:

[0075] If the hardware is in a low-load state, set the selection range of the convolution module structure determination result to the low-load convolution module structure determination result, and set the selection range of the channel pruning state determination result to the low-load channel pruning state determination result.

[0076] If the hardware state is medium load, set the selection range of the convolution module structure determination result to the medium load convolution module structure determination result, and set the selection range of the channel pruning state determination result to the medium load channel pruning state determination result.

[0077] If the hardware is under high load, set the selection range of the convolution module structure determination result to the high load convolution module structure determination result selection range, and set the selection range of the channel pruning state determination result to the high load channel pruning state determination result selection range.

[0078] In one optional embodiment of this application, such as Figure 2 As shown, deep learning-based image processing methods may also include:

[0079] Step S208: If the final convolution module structure determination result is the basic convolution parallel depth-separable convolution module structure determination result, set the convolution module of the image processing deep learning model corresponding to the image to be processed as the basic convolution parallel depth-separable convolution module.

[0080] Step S209: If the final convolutional module structure determination result is the basic convolutional module structure determination result, set the convolutional module of the image processing deep learning model corresponding to the image to be processed as the basic convolutional module.

[0081] Step S210: If the final convolutional module structure determination result is the depth-separable convolutional module structure determination result, set the convolutional module of the image processing deep learning model corresponding to the image to be processed as a depth-separable convolutional module.

[0082] Step S211: If the final channel pruning state determination result is a reduced pruning degree state determination result, set the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed to a reduced pruning degree channel pruning strategy.

[0083] Step S212: If the final channel pruning state determination result is the normal pruning degree state determination result, set the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed to the normal pruning degree channel pruning strategy.

[0084] Step S213: If the final channel pruning state determination result is an increased pruning degree state determination result, set the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed to an increased pruning degree channel pruning strategy.

[0085] In an optional embodiment of this application, the convolutional module structure determination result and the channel pruning state determination result are selected from the selection range of the convolutional module structure determination result and the selection range of the channel pruning state determination result, respectively, based on image complexity and inter-frame difference. This may include:

[0086] Based on image complexity and inter-frame difference, and combining the convolutional module structure determination weight and pruning degree determination weight, the required values ​​for convolutional module structure complexity and pruning degree model complexity are calculated respectively. Specifically, the convolutional module structure determination weight for image complexity is greater than that for inter-frame difference, while the pruning degree determination weight for image complexity is less than that for inter-frame difference.

[0087] The convolutional module structure determination result with the smallest difference between the actual value of the convolutional module structure complexity and the required value of the convolutional module structure complexity within the range of convolutional module structure determination results is selected as the final convolutional module structure determination result.

[0088] The channel pruning state determination result with the smallest difference between the actual value of the pruning degree model complexity and the required value of the pruning degree model complexity within the selected range of channel pruning state determination results is selected as the final channel pruning state determination result.

[0089] In an optional embodiment of this application, the hardware resource status parameters include computing resource utilization and memory bandwidth utilization. The hardware status value can be calculated by weighted summation of computing resource utilization and memory bandwidth utilization.

[0090] In an optional embodiment of this application, the expressions for image complexity and inter-frame difference can be:

[0091]

[0092]

[0093] In the formula, and The video frame number is determined by the image complexity and the video frame number, respectively. Inter-frame differences and These are the pixel height and pixel width of the image to be processed, respectively. and These are the texture complexity weights and gradient complexity weights, respectively. For the image to be processed at the pixel point Local binary pattern value at that location, and The image to be processed is located at the pixel level. The horizontal and vertical gradient values ​​are obtained by the Sobel operator. This represents the total number of video frames within the time window for inter-frame difference analysis. The video frame number is The image to be processed at the pixel point Pixel value at that location, The video frame number is The image to be processed at the pixel point The pixel value at that location.

[0094] To illustrate, Local Binary Pattern (LBP) is an operator used to describe local texture features of an image. The image processing terminal can calculate LBP for each pixel according to its basic calculation rules. Centered on the pixel, a 3×3 neighborhood window is selected. The grayscale value of each pixel within the neighborhood is compared with the grayscale value of the center pixel. If the grayscale value of a neighboring pixel is greater than or equal to the grayscale value of the center pixel, it is marked as 1; otherwise, it is marked as 0. These binary numbers are then arranged in a clockwise or counterclockwise direction to form an 8-bit binary number. The decimal value corresponding to this binary number is the value of the pixel. LBP value. During the calculation process, for pixels at the image edge, their 3×3 neighborhood may extend beyond the image boundary. The image processing terminal can use methods such as mirror filling or zero filling to supplement boundary pixels, ensuring the integrity of the LBP value calculation.

[0095] Optionally, the image processing terminal needs to calculate each pixel. The horizontal gradient value after processing by the Sobel operator and vertical gradient value The Sobel operator is a commonly used edge detection operator. Image processing terminals can perform convolution operations with the image to be processed using horizontal and vertical convolution kernels respectively. The horizontal convolution kernel is used to detect vertical edges, and the vertical convolution kernel is used to detect horizontal edges. Through convolution operations, the gradient intensity of each pixel in both directions can be obtained, i.e. and During the calculation process, the image processing terminal can first convert the image to be processed into a grayscale image, and then perform a convolution operation on the grayscale image to ensure the accuracy of gradient calculation.

[0096] Indicative of the total number of video frames in the inter-frame difference analysis time window. The value needs to take into account both the video frame rate and the real-time processing requirements, and can usually be set to 2 to 10 frames.

[0097] As an illustration, the image processing terminal can determine the sequence number of the current video frame to be processed. Then, extract the sequence number from the image sequence to be processed. and The image to be processed, for example, when =10, When =5, The numbers 1, 2, 3, 4 and 5 can be selected respectively, and the corresponding frame number combinations to be compared are (6,5), (7,6), (8,7), (9,8) and (10,9). By comparing the images to be processed in pairs of these consecutive frames, the inter-frame changes in the period before the current frame can be fully reflected.

[0098] In one exemplary embodiment of this application, such as Figure 2 As shown, a deep learning-based image processing method is provided, which may include:

[0099] Step S201: Obtain the dynamic video stream to be processed.

[0100] Step S202: Obtain the hardware resource status parameters corresponding to the video frames of the images to be processed, and calculate the corresponding hardware status values ​​of each image to be processed based on the hardware resource status parameters.

[0101] Step S203: Calculate the image complexity and inter-frame difference of the image to be processed.

[0102] Step S204: Based on the hardware status value and combined with the preset hardware status threshold range, the hardware status corresponding to the image to be processed is divided.

[0103] Step S205: Based on the hardware status, set the selection range of the convolution module structure determination result and the selection range of the channel pruning status determination result.

[0104] Step S206: Based on the image complexity and inter-frame difference, the final convolutional module structure determination result and the final channel pruning state determination result are selected from the selection range of the convolutional module structure determination result and the selection range of the channel pruning state determination result, respectively.

[0105] Step S207: Summarize the final convolutional module structure determination results and the final channel pruning state determination results to generate the image processing deep learning model state determination results.

[0106] Step S208: If the final convolution module structure determination result is the basic convolution parallel depth-separable convolution module structure determination result, set the convolution module of the image processing deep learning model corresponding to the image to be processed as the basic convolution parallel depth-separable convolution module.

[0107] Step S209: If the final convolutional module structure determination result is the basic convolutional module structure determination result, set the convolutional module of the image processing deep learning model corresponding to the image to be processed as the basic convolutional module.

[0108] Step S210: If the final convolutional module structure determination result is the depth-separable convolutional module structure determination result, set the convolutional module of the image processing deep learning model corresponding to the image to be processed as a depth-separable convolutional module.

[0109] Step S211: If the final channel pruning state determination result is a reduced pruning degree state determination result, set the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed to a reduced pruning degree channel pruning strategy.

[0110] Step S212: If the final channel pruning state determination result is the normal pruning degree state determination result, set the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed to the normal pruning degree channel pruning strategy.

[0111] Step S213: If the final channel pruning state determination result is an increased pruning degree state determination result, set the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed to an increased pruning degree channel pruning strategy.

[0112] The aforementioned deep learning-based image processing method dynamically and specifically adjusts the convolutional module structure and channel pruning strategy of the deep learning model for image processing by comprehensively considering the hardware resource status, image complexity, and inter-frame differences of each frame in the dynamic video stream to be processed. This enables hardware adaptation and dynamic model architecture fine-tuning, significantly improving the model inference efficiency, image processing quality, and hardware resource utilization in video stream processing.

[0113] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0114] Based on the same inventive concept, this application also provides a deep learning-based image processing apparatus for implementing the deep learning-based image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more deep learning-based image processing apparatus embodiments provided below can be found in the limitations of the deep learning-based image processing method described above, and will not be repeated here.

[0115] In one exemplary embodiment, such as Figure 3 As shown, a deep learning-based image processing device 300 is provided, which may include:

[0116] The image acquisition module 301 can be used to acquire the dynamic video stream to be processed, which includes the images to be processed for each video frame.

[0117] The hardware status monitoring module 302 can be used to obtain the hardware resource status parameters corresponding to the video frames of the image to be processed, and calculate the corresponding hardware status value of each image to be processed based on the hardware resource status parameters.

[0118] The image parameter calculation module 303 can be used to calculate the image complexity and inter-frame difference of the image to be processed.

[0119] The model parameter adjustment module 304 can be used to determine the state of the deep learning model for image processing of the image to be processed based on hardware state values, image complexity, and inter-frame differences, and generate the state determination result of the deep learning model for image processing of the image to be processed.

[0120] In an optional embodiment of this application, the graph model parameter adjustment module 304 can also be used for:

[0121] Based on the hardware status values ​​and combined with the preset hardware status threshold range, the hardware status corresponding to the image to be processed is determined.

[0122] The selection range for the convolution module structure determination result and the selection range for the channel pruning state determination result are set based on the hardware status.

[0123] Based on image complexity and inter-frame difference, the final convolutional module structure determination result and the final channel pruning state determination result are selected from the range of convolutional module structure determination result and the range of channel pruning state determination result, respectively.

[0124] The final convolutional module structure determination results and the final channel pruning state determination results are summarized to generate the image processing deep learning model state determination results.

[0125] In an optional embodiment of this application, the graph model parameter adjustment module 304 can also be used for:

[0126] If the hardware is in a low-load state, set the selection range of the convolution module structure determination result to the low-load convolution module structure determination result, and set the selection range of the channel pruning state determination result to the low-load channel pruning state determination result.

[0127] If the hardware state is medium load, set the selection range of the convolution module structure determination result to the medium load convolution module structure determination result, and set the selection range of the channel pruning state determination result to the medium load channel pruning state determination result.

[0128] If the hardware is under high load, set the selection range of the convolution module structure determination result to the high load convolution module structure determination result selection range, and set the selection range of the channel pruning state determination result to the high load channel pruning state determination result selection range.

[0129] In an optional embodiment of this application, the deep learning-based image processing apparatus 300 can also be used for:

[0130] If the final convolutional module structure determination result is the basic convolutional parallel depthwise separable convolutional module structure determination result, then the convolutional module of the image processing deep learning model corresponding to the image to be processed is set as the basic convolutional parallel depthwise separable convolutional module.

[0131] If the final convolutional module structure determination result is the basic convolutional module structure determination result, then the convolutional module of the image processing deep learning model corresponding to the image to be processed is set as the basic convolutional module.

[0132] If the final convolutional module structure determination result is a depth-separable convolutional module structure determination result, then the convolutional module of the image processing deep learning model corresponding to the image to be processed is set as a depth-separable convolutional module.

[0133] If the final channel pruning state determination result is a reduced pruning degree state determination result, the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed is set to the reduced pruning degree channel pruning strategy.

[0134] If the final channel pruning state determination result is a normal pruning degree state determination result, the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed will be set to the normal pruning degree channel pruning strategy.

[0135] If the final channel pruning state determination result is an increased pruning degree state determination result, the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed will be set to the increased pruning degree channel pruning strategy.

[0136] In an optional embodiment of this application, the graph model parameter adjustment module 304 can also be used for:

[0137] Based on image complexity and inter-frame difference, and combining the convolutional module structure determination weight and pruning degree determination weight, the required values ​​for convolutional module structure complexity and pruning degree model complexity are calculated respectively. Specifically, the convolutional module structure determination weight for image complexity is greater than that for inter-frame difference, while the pruning degree determination weight for image complexity is less than that for inter-frame difference.

[0138] The convolutional module structure determination result with the smallest difference between the actual value of the convolutional module structure complexity and the required value of the convolutional module structure complexity within the range of convolutional module structure determination results is selected as the final convolutional module structure determination result.

[0139] The channel pruning state determination result with the smallest difference between the actual value of the pruning degree model complexity and the required value of the pruning degree model complexity within the selected range of channel pruning state determination results is selected as the final channel pruning state determination result.

[0140] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a deep learning-based image processing method as described above.

[0141] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0142] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0143] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A deep learning-based image processing method, characterized in that, The method includes: Acquire a dynamic video stream to be processed, wherein the dynamic video stream to be processed includes images of each video frame to be processed; Obtain the hardware resource status parameters corresponding to the video frames of the image to be processed, and calculate the corresponding hardware status values ​​of each image to be processed based on the hardware resource status parameters. Calculate the image complexity and inter-frame difference of the image to be processed; Based on the hardware status value, the image complexity, and the inter-frame difference, the image processing deep learning model status determination is performed on the image to be processed, and the image processing deep learning model status determination result of the image to be processed is generated.

2. The method according to claim 1, characterized in that, The image processing deep learning model state determination result includes convolutional module structure determination result, channel pruning state determination result, and precision quantization determination result. The process of determining the image processing deep learning model state based on the hardware state value, the image complexity, and the inter-frame difference, and generating the image processing deep learning model state determination result, includes: Based on the hardware status value and combined with the preset hardware status threshold range, the hardware status corresponding to the image to be processed is obtained. Based on the hardware state settings, the selection range of the convolution module structure determination result and the selection range of the channel pruning state determination result are set. Based on the image complexity and the inter-frame difference, the final convolutional module structure determination result and the final channel pruning state determination result are selected from the selection range of the convolutional module structure determination result and the selection range of the channel pruning state determination result, respectively. The final convolutional module structure determination results and the final channel pruning state determination results are combined to generate the image processing deep learning model state determination results.

3. The method according to claim 2, characterized in that, The hardware states include low load, medium load, and high load states. The convolutional module structure determination results include the structure determination results of basic convolutional parallel depthwise separable convolutional modules, basic convolutional module structure determination results, and depthwise separable convolutional module structure determination results. The channel pruning state determination results include the determination results of reduced pruning degree, normal pruning degree, and increased pruning degree. The selection range of the convolutional module structure determination results includes the selection range of low load convolutional module structure determination results, medium load convolutional module structure determination results, and high load convolutional module structure determination results. The selection range of the channel pruning state determination results includes the selection range of low load channel pruning state determination results, medium load channel pruning state determination results, and high load channel pruning state determination results. The selection range of the low-load convolution module structure determination result includes the structure determination result of the basic convolution parallel depth-separable convolution module and the structure determination result of the basic convolution module; the selection range of the medium-load convolution module structure determination result includes the structure determination result of the basic convolution module and the structure determination result of the depth-separable convolution module; and the selection range of the high-load convolution module structure determination result includes the structure determination result of the depth-separable convolution module. The selection range of the low-load channel pruning status determination result includes the reduced pruning degree status determination result and the normal pruning degree status determination result; the selection range of the medium-load channel pruning status determination result includes the normal pruning degree status determination result and the increased pruning degree status determination result; and the selection range of the high-load channel pruning status determination result includes the increased pruning degree status determination result. The selection range for the convolution module structure determination result and the selection range for the channel pruning state determination result based on the hardware state settings include: If the hardware state is the low-load state, the selection range of the convolution module structure determination result is set to the selection range of the low-load convolution module structure determination result, and the selection range of the channel pruning state determination result is set to the selection range of the low-load channel pruning state determination result. If the hardware state is the medium load state, the selection range of the convolution module structure determination result is set to the selection range of the medium load convolution module structure determination result, and the selection range of the channel pruning state determination result is set to the selection range of the medium load channel pruning state determination result. If the hardware state is the high-load state, the selection range of the convolution module structure determination result is set to the selection range of the high-load convolution module structure determination result, and the selection range of the channel pruning state determination result is set to the selection range of the high-load channel pruning state determination result.

4. The method according to claim 3, characterized in that, The method further includes: If the final convolutional module structure determination result is the same as the basic convolutional parallel depth-separable convolutional module structure determination result, the convolutional module of the image processing deep learning model corresponding to the image to be processed is set as the basic convolutional parallel depth-separable convolutional module. If the final convolutional module structure determination result is the same as the basic convolutional module structure determination result, the convolutional module of the image processing deep learning model corresponding to the image to be processed is set as the basic convolutional module. If the final convolutional module structure determination result is the depth-separable convolutional module structure determination result, the convolutional module of the image processing deep learning model corresponding to the image to be processed is set as a depth-separable convolutional module. If the final channel pruning state determination result is the reduced pruning degree state determination result, the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed is set to the reduced pruning degree channel pruning strategy. If the final channel pruning state determination result is the normal pruning degree state determination result, the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed is set to the normal pruning degree channel pruning strategy. If the final channel pruning state determination result is the increased pruning degree state determination result, the channel pruning strategy of the image processing deep learning model corresponding to the image to be processed is set to the increased pruning degree channel pruning strategy.

5. The method according to claim 3, characterized in that, The step of selecting the convolutional module structure determination result and the channel pruning state determination result from the selection range of the convolutional module structure determination result and the selection range of the channel pruning state determination result based on the image complexity and the inter-frame difference includes: Based on the image complexity and the inter-frame difference, and combined with the convolutional module structure determination weight and the pruning degree determination weight, the required values ​​for the convolutional module structure complexity and the pruning degree model complexity are calculated respectively; wherein, the convolutional module structure determination weight for the image complexity is greater than the convolutional module structure determination weight for the inter-frame difference, and the pruning degree determination weight for the image complexity is less than the pruning degree determination weight for the inter-frame difference. The convolutional module structure determination result whose difference between the actual value of the convolutional module structure complexity and the required value of the convolutional module structure complexity is the smallest among the convolutional module structure determination results within the range of the convolutional module structure determination results is selected as the final convolutional module structure determination result. The channel pruning state determination result whose difference between the actual value of the pruning degree model complexity and the required value of the pruning degree model complexity is the smallest among the channel pruning state determination results is selected as the final channel pruning state determination result.

6. The method according to any one of claims 1 to 5, characterized in that, The hardware resource status parameters include computing resource utilization and memory bandwidth utilization. The hardware status value is calculated by weighted summation of the computing resource utilization and the memory bandwidth utilization.

7. The method according to any one of claims 1 to 5, characterized in that, The expressions for the image complexity and the inter-frame difference are: In the formula, and The image complexity and the video frame number are respectively: The inter-frame difference, and These are the pixel height and pixel width of the image to be processed, respectively. and These are the texture complexity weights and gradient complexity weights, respectively. For the image to be processed at the pixel point Local binary pattern value at that location, and The image to be processed is located at the pixel points. The horizontal and vertical gradient values ​​are obtained by the Sobel operator. This represents the total number of video frames within the inter-frame difference analysis time window. The video frame number is The image to be processed at pixel points Pixel value at that location, The video frame number is The image to be processed at pixel points The pixel value at that location.

8. An image processing device based on deep learning, characterized in that, The device includes: The image acquisition module is used to acquire the dynamic video stream to be processed, which includes the images to be processed for each video frame. The hardware status monitoring module is used to obtain the hardware resource status parameters corresponding to the video frames of the image to be processed, and calculate the corresponding hardware status value of each image to be processed based on the hardware resource status parameters. An image parameter calculation module is used to calculate the image complexity and inter-frame difference of the image to be processed. The model parameter adjustment module is used to determine the state of the deep learning model for image processing of the image to be processed based on the hardware state value, the image complexity, and the inter-frame difference, and to generate the state determination result of the deep learning model for image processing of the image to be processed.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images