Image data expansion method and system

Abstract

The present invention provides an image data expansion method and system. The method obtains an initial image, inputs the initial image into a denoising network, and obtains a denoised image; uses a correction function and an image processing filter function to enhance the brightness, merges the processed brightness with the hue and the saturation, obtains a merged color space, and reversely converts the color space into a brightened image; uses a single-level feature pyramid structure to extract a fused image of the brightened image, uses an encoder that introduces a deep convolutional layer to extract local features of the fused image, aggregates the local features to form a local fused image, and uses a decoder to decode the local fused image to obtain a target image; uses a segmentation model to segment the target of the target image, and obtains a mask of the target to obtain an expanded image, thereby effectively obtaining a number of fine data.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image data expansion method and system. Background Art

[0002] Currently, in the field of data expansion and image enhancement, especially in the field of industrial image data expansion and data enhancement, there are many difficulties in collecting real data, such as scarcity of real data, high collection costs, and the inability to collect a large amount of comprehensive real data in a short period of time. This has a great impact on the subsequent image processing and model training work.

[0003] Existing image enhancement methods are difficult to obtain good and detailed enhancement effects in view of the above difficulties. For example, an image segmenter is used as a pre-target acquisition component, and the mask of the acquired image target is enhanced into the real scene image by copying and pasting. There are difficulties such as cumbersome preliminary annotation work, low adaptability when adding new categories, and the need to train the image segmentation model separately. When using a target detector as a pre-target acquisition component, since the target detection result is usually a rectangle, there is redundant image information in the acquired target detection frame, and it is impossible to accurately deduct the target image. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an image data expansion method and system for solving the technical problems in the prior art.

[0005] On the one hand, the invention provides the following technical solution, a method for image data expansion, the method comprising:

[0006] Acquire an initial image, input the initial image into an estimation network to obtain an estimated value of the initial image, input the estimated value and the initial image into a denoising network to obtain a denoised image;

[0007] transforming the denoised image to obtain the hue, saturation, and brightness of the denoised image, enhancing the brightness using a correction function and an image processing filter function, merging the processed brightness with the hue and the saturation to obtain a merged color space, and reversely transforming the color space into a brightened image;

[0008] A fused image of the brightened image is extracted using a single-level feature pyramid structure, local features of the fused image are extracted using an encoder that introduces a deep convolutional layer, the local features are aggregated to form a local fused image, and a decoder is used to decode the local fused image to obtain a target image;

[0009] The target of the target image is segmented by using a segmentation model, and a mask of the target is obtained. The mask is pasted, mirrored, and scaled on the image to be expanded based on the mask by using an image processing library to obtain an expanded image.

[0010] Compared with the prior art, the beneficial effects of the present application are as follows: by obtaining the denoised image and the brightened image, the noise of the image can be effectively removed and the brightness of the image can be improved, so as to facilitate obtaining a high-precision target image in the later stage; and then by the step of obtaining the target image, the detection capability of targets of different sizes can be improved, so that more accurate targets can be obtained and more refined operations can be achieved; finally, by the step of obtaining the expanded image, a number of fine data can be effectively obtained.

[0011] Furthermore, the step of processing the brightness using a correction function and an image processing filter function includes:

[0012] Performing image processing on the brightness to obtain a first processing layer and a second processing layer;

[0013] Extracting the illumination component of the first processing layer by using an image processing filter function, and calculating based on the illumination component by using a weighted calculation formula to obtain a reflection layer;

[0014] The second processing layer is searched by the algorithm to obtain a correction value, and a correction function is used to correct the brightness component of each pixel point of the second processing layer based on the correction value to form a brightness layer;

[0015] The brightness layer is superimposed on the reflection layer, and fusion compensation is performed to obtain the illumination layer.

[0016] Furthermore, the weighted calculation formula includes:

[0017]

[0018] Represented as the reflection layer, represents the brightness of the ith channel at position (x,y), Represented as a convolution operation, Expressed as an image processing filter function, Expressed as weight coefficient, The scale of the convolution kernel, Respectively represent the horizontal and vertical coordinates of the pixel;

[0019] The correction function comprises:

[0020]

[0021] in, represents the output image, Expressed as the correction value, It is represented as pixel gray value, and q is represented as correction parameter.

[0022] Furthermore, the estimation network includes five convolutional neural network frameworks and a moving window mechanism;

[0023] The denoising network includes two residual structures, two convolutional neural network frameworks and three multi-scale structures, wherein the multi-scale structure includes a branch unit and a fusion unit.

[0024] Furthermore, the step of extracting the fused image of the brightened image using a single-level feature pyramid structure includes:

[0025] The downsampling operation in the residual network is replaced by a dilated convolution group with different dilation coefficients, and multiple groups of parallel transposed convolution operations are used on its last layer output to form a single-level feature pyramid structure.

[0026] The single-level feature pyramid structure is used to extract a feature image containing shallow and deep feature information of the brightened image, and transposed convolution operations of different sizes are performed on the feature image to form a fused image.

[0027] Furthermore, the encoder that introduces the deep convolution layer includes: an input embedding layer, a multi-head attention layer, a residual normalization layer, a deep convolution layer and a fully connected layer, wherein the deep convolution layer is used to enhance the relevance of local information of the context.

[0028] Furthermore, a weight-constrained attention mechanism is added to the decoder;

[0029] The attention mechanism includes:

[0030]

[0031] in, Indicates x The first x The output of the layer, , and Respectively represent x The query of each header, the corresponding value and key, represents the bias term of the residual connection, x is the index value of the layer number, T represents transposition, Represents the output of the previous layer.

[0032] In a second aspect, the invention provides the following technical solution: an image data expansion system, the system comprising:

[0033] A denoising module is used to obtain an initial image, input the initial image into an estimation network to obtain an estimated value of the initial image, and input the estimated value and the initial image into a denoising network to obtain a denoised image;

[0034] a brightening module, for transforming the denoised image to obtain the hue, saturation, and brightness of the denoised image, enhancing the brightness using a correction function and an image processing filter function, merging the processed brightness with the hue and the saturation to obtain a merged color space, and reversely transforming the color space into a brightened image;

[0035] A decoding module, used for extracting a fused image of the brightened image using a single-level feature pyramid structure, extracting local features of the fused image using an encoder that introduces a deep convolutional layer, aggregating the local features to form a local fused image, and decoding the local fused image using a decoder to obtain a target image;

[0036] The expansion module is used to segment the target of the target image using the segmentation model and obtain the mask of the target, and use the image processing library to paste, mirror, and scale the mask on the image to be expanded based on the mask to obtain the expanded image.

[0037] In a third aspect, the invention provides the following technical solution: a computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image data expansion method as described above when executing the computer program.

[0038] In a fourth aspect, the invention provides the following technical solution: a storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned image data expansion method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A flowchart of an image data expansion method provided by a first embodiment of the present invention;

[0041] Figure 2 A schematic diagram of the structure of the connection between the estimation network and the denoising network of the image data expansion method provided by the first embodiment of the present invention;

[0042] Figure 3 A schematic diagram of the structure of a multi-scale structure of an image data expansion method provided by the first embodiment of the present invention;

[0043] Figure 4 A schematic diagram of the structure of an encoder introducing a deep convolutional layer in the image data expansion method provided by the first embodiment of the present invention;

[0044] Figure 5 A structural block diagram of an image data expansion system provided by a second embodiment of the present invention;

[0045] Figure 6 A schematic diagram of the hardware structure of a computer provided in the third embodiment of the present invention.

[0046] The embodiments of the present invention will be further described below with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout are the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the embodiments of the present invention, and should not be construed as limiting the present invention.

[0048] In the description of the embodiments of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0049] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0050] Embodiment 1

[0051] In the first embodiment of the present invention, see Figure 1 , an image data expansion method, comprising the following steps S01 to S04:

[0052] S01, obtaining an initial image, inputting the initial image into an estimation network to obtain an estimated value of the initial image, and inputting the estimated value and the initial image into a denoising network to obtain a denoised image;

[0053] The initial image contains the desired target, for example, the target may be a screw on a joist in a stereoscopic warehouse, a screw, a nut, a wrench, etc. The initial image is obtained using a web crawler technology.

[0054] See also Figure 2 and Figure 3 ,Specifically, the estimation network includes five convolutional neural network frameworks and a ,moving window mechanism;

[0055] The denoising network includes two residual structures, two convolutional neural network frameworks and three multi-scale structures, wherein the multi-scale structure includes a branch unit and a fusion unit, wherein the branch unit includes four 3×3 convolutional layers (hole convolution) and a pooling layer, and the fusion unit includes a 1×1 convolutional layer and a residual block formed by two 3×3 convolutional layers. The branch unit is used to decompose the features into five features, and the fusion unit fuses the decomposed five features in the form of residual connection. Through the cooperation of the fusion unit and the branch unit, not only can a wide range of feature information be captured through hole convolution, but also key local features can be retained. The output of the estimation network corresponds to the input of the denoising network.

[0056] In this embodiment, the convolutional neural network framework includes a 3x3 convolution and a ReLU (Rectified Linear Unit), and the moving window mechanism includes two 1x1 convolutions and a swin-conv structure (Swin Transformer Convolution), wherein the first four convolutional neural network frameworks are connected in sequence, and the first four convolutional neural network frameworks are connected to the moving window mechanism after being connected, and the moving window mechanism is connected to the fifth convolutional neural network framework;

[0057] The three multi-scale structures, the two residual structures and the two convolutional neural network frameworks are connected in sequence, and the front-end multi-scale structure is connected to the fifth convolutional neural network framework so that the output of the estimation network becomes the input of the denoising network.

[0058] It is worth noting that the swin-conv structure (Swin Transformer Convolution) is a key component in the Swin Transformer architecture. It is an adaptive convolution variant used to enhance the model's ability to capture features of different scales, thereby promoting the ability to extract features. The swin-conv structure performs a self-attention mechanism adjustment on the data features obtained from the noise level estimate, where padding is performed on each 3x3 convolution in order to maintain the consistency of the image size during the 3x3 convolution process. By using a residual structure, specific noise patterns in the image can be more effectively suppressed, thereby improving the overall performance of the denoising effect. Each residual structure is composed of two convolutional layers. After each convolution process, the ReLU activation function is implemented. Before the second ReLU activation is implemented, the output of this layer needs to be accumulated with the initial input to form a complete residual structure.

[0059] S02, transforming the denoised image to obtain the hue, saturation, and brightness of the denoised image, enhancing the brightness using a correction function and an image processing filter function, merging the processed brightness with the hue and the saturation to obtain a merged color space, and reversely transforming the color space into a brightened image;

[0060] Specifically, the step of processing the brightness using a correction function and an image processing filter function includes:

[0061] S21, performing image processing on the brightness to obtain a first processing layer and a second processing layer;

[0062] S22, extracting the illumination component of the first processing layer by using an image processing filter function, and obtaining a reflection layer by using a weighted calculation formula based on the illumination component;

[0063] S23, using the search algorithm for the second processing layer to obtain a correction value, and using a correction function to correct the brightness component of each pixel point in the second processing layer based on the correction value to form a brightness layer;

[0064] S24, superimposing the brightness layer onto the reflection layer, and performing fusion compensation to obtain an illumination layer.

[0065] After obtaining the illumination layer, the illumination layer is combined with the hue and the saturation to obtain an enhanced HSV, and the enhanced HSV is reversely converted into the brightened RGB, that is, the illumination layer (brightened V) is combined with the H and S components, and then converted back to RGB to obtain the final illumination-compensated brightened image, wherein the illumination layer is the processed brightness, and the brightened RGB is the brightened image.

[0066] In this embodiment, the denoised image is first converted from RGB (Red Green Blue) space to HSV (Hue Saturation Value) space, and then V (brightness) is enhanced to ensure that the processing of the V component does not affect the other two components.

[0067] Optionally, the image processing filter function may be a Gaussian kernel function (GKF) or a fast guided filter function (FGF). In this embodiment, the image processing filter function is a fast guided filter function. The fast guided filter function not only reduces the computational complexity, but also avoids image distortion and improves the detail expression capability. Optionally, the correction function may be a two-dimensional gamma function.

[0068] The expression of the brightness layer is:

[0069]

[0070] in, Represented as the brightness layer, Represented as the second processing layer.

[0071] More specifically, the weighted calculation formula includes:

[0072]

[0073] Represented as the reflection layer, represents the brightness of the ith channel at position (x,y), Represented as a convolution operation, Expressed as an image processing filter function, Expressed as weight coefficient, The scale of the convolution kernel, Respectively represent the horizontal and vertical coordinates of the pixel;

[0074] The correction function comprises:

[0075]

[0076] in, represents the output image, Expressed as the correction value, It is represented as the pixel gray value, and q is represented as the correction parameter, whose value is generally 2.5.

[0077] In this embodiment, The parameter value range is 0~1. Approaching zero, we get maximum illumination compensation. Approaching 1, the light intensity is basically unchanged. It can be seen that it is most critical to have a suitable value for the correction value, so a search algorithm is used for searching. The search algorithm is the Sparrow Search Algorithm (SSA), which is a meta-heuristic optimization algorithm that simulates the foraging behavior of sparrows in nature. This algorithm is usually used to solve global optimization problems, especially those with complex search spaces. The basic idea of ​​the sparrow search algorithm is to imitate the social behavior and foraging strategy of sparrows when looking for food. Among them, the optimal value of the correction value obtained by testing the initial image is 0.985.

[0078] S03, extracting a fused image of the brightened image using a single-level feature pyramid structure, extracting local features of the fused image using an encoder that introduces a deep convolutional layer, aggregating the local features to form a local fused image, and decoding the local fused image using a decoder to obtain a target image;

[0079] Specifically, the step of extracting the fused image of the brightened image using a single-level feature pyramid structure includes:

[0080] S31, uses dilated convolution groups with different dilation coefficients to replace the downsampling operation in the residual network, and uses multiple sets of parallel transposed convolution operations on its last layer output to form a single-level feature pyramid structure;

[0081] S32, extracting a feature image containing shallow and deep feature information of the brightened image by using the single-level feature pyramid structure, and performing transposed convolution operations of different sizes on the feature image to form a fused image.

[0082] Among them, the residual network is Resnet-50. Dilated Convolution Group is a special convolution operation, which is characterized by the gradual increase of the coefficient of the convolution kernel (i.e., the dilation rate). The characteristic of the dilated convolution group is that its dilation rate increases layer by layer. The dilation rate of the first layer is the base value, the dilation rate of the second layer is 3, and the dilation rate of the third layer is 6. This means that as the number of layers increases, the range of the input feature map covered by the convolution kernel is also increasing. Through dilated convolution, the model can obtain a feature map with a continuous receptive field. This continuous receptive field helps to match feature changes of different scales. Single-level Feature Pyramid is a concept in computer vision, mainly used for image recognition and target detection tasks. The core idea of ​​the feature pyramid is to capture multi-scale information in the image through feature maps of different resolutions, thereby improving the model's ability to detect targets of different sizes.

[0083] It is worth noting that by using a dilated convolution group with different dilation coefficients to replace the downsampling operation steps in the residual network, the loss of detail information in the downsampling process is prevented, thereby improving the performance in small-size target detection; then, by using multiple groups of parallel transposed convolution operations, the feature map is restored to achieve multi-scale fusion of features.

[0084] See also Figure 4 Specifically, the encoder that introduces the deep convolution layer includes: an input embedding layer, a multi-head attention layer, a residual normalization layer, a deep convolution layer and a fully connected layer, wherein the deep convolution layer is used to enhance the relevance of local context information, and the residual normalization layer includes residual connection (Residual Connection) and normalization (Normalization).

[0085] Among them, the encoder that introduces the deep convolution layer is the encoder in the Transformer model. The deep convolution can not only reduce the number of parameters of the Transformer model, but also realize the extraction and aggregation of local information of the target features (features of the fused image). By using deep convolution in the fully connected layer and combining the self-attention mechanism, the association between global features and local features can be established on the image, which helps to improve the accuracy of the model in fine recognition tasks. Deep convolution processes spatial information in the length and width directions by performing convolution operations on each channel of the feature. Point-by-point convolution (1x1 convolution) is used to achieve cross-channel information integration, maintain the dimension of the output feature map, and reduce the amount of calculation of the model.

[0086] Furthermore, a weight-constrained attention mechanism is added to the decoder;

[0087] The attention mechanism includes:

[0088]

[0089] in, Indicates x The first x The output of the layer, , and Respectively represent x The query of each header, the corresponding value and key, represents the bias term of the residual connection, x is the index value of the layer number, T represents transposition, Represents the output of the previous layer.

[0090] It is worth noting that by adding a weight-constrained attention mechanism to the attention part of the decoder, the global attention calculation is limited to a local range, thereby reducing the amount of calculation.

[0091] S04, using the segmentation model to segment the target of the target image, and obtaining the mask of the target, and using the image processing library to paste, mirror, and scale the mask on the image to be expanded based on the mask to obtain an expanded image.

[0092] In this embodiment, the image processing library may be OpenCV (Open Source Computer Vision Library), which is an open source computer vision and machine learning software library. The segmentation model is the DeepLab v3+ model, which is a deep learning model for semantic image segmentation and is a further development of the DeepLab series of algorithms. It achieves pixel-level semantic fine segmentation.

[0093] In summary, an image data expansion method can effectively remove the noise of the image and improve the brightness of the image through the steps of obtaining a denoised image and a brightened image, so as to facilitate the later acquisition of a high-precision target image. Then, through the step of obtaining the target image, the detection capability of targets of different sizes is improved, so that more precise targets can be obtained and more refined operations can be achieved. Finally, through the step of obtaining an expanded image, a number of fine data can be effectively obtained.

[0094] Embodiment 2

[0095] like Figure 5 As shown, in a second embodiment of the present invention, an image data expansion system is provided, the system comprising:

[0096] The denoising module 10 is used to obtain an initial image, input the initial image into an estimation network to obtain an estimated value of the initial image, and input the estimated value and the initial image into a denoising network to obtain a denoised image;

[0097] A brightening module 20 is used to transform the denoised image to obtain the hue, saturation, and brightness of the denoised image, enhance the brightness using a correction function and an image processing filter function, merge the processed brightness with the hue and the saturation to obtain a merged color space, and reversely convert the color space into a brightened image;

[0098] A decoding module 30 is used to extract a fused image of the brightened image using a single-level feature pyramid structure, extract local features of the fused image using an encoder that introduces a deep convolutional layer, aggregate the local features to form a local fused image, and decode the local fused image using a decoder to obtain a target image;

[0099] The expansion module 40 is used to segment the target of the target image using the segmentation model and obtain the mask of the target, and use the image processing library to paste, mirror, and scale the mask on the image to be expanded based on the mask to obtain an expanded image.

[0100] The image data expansion system provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0101] Embodiment 3

[0102] like Figure 6 As shown, in the third embodiment of the present invention, the embodiment of the present invention provides the following technical solution, a computer, including a memory 202, a processor 201, and a computer program stored in the memory 202 and executable on the processor 201, and the processor 201 implements the image data expansion method as described above when executing the computer program.

[0103] Specifically, the processor 201 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0104] Among them, the memory 202 may include a large-capacity memory for data or instructions. For example, but not limitation, the memory 202 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 202 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 202 may be inside or outside the data processing device. In a specific embodiment, the memory 202 is a non-volatile memory. In a specific embodiment, the memory 202 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0105] The memory 202 may be used to store or cache various data files that need to be processed and / or used for communication, as well as possible computer program instructions executed by the processor 201 .

[0106] The processor 201 implements the above-mentioned image data expansion method by reading and executing the computer program instructions stored in the memory 202 .

[0107] In some embodiments, the computer may further include a communication interface 203 and a bus 200. Figure 6 As shown, the processor 201, the memory 202, and the communication interface 203 are connected via a bus 200 and communicate with each other.

[0108] The communication interface 203 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication interface 203 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0109] The bus 200 includes hardware, software or both, and couples the components of the computer to each other. The bus 200 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 200 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 200 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0110] Embodiment 4

[0111] In a fourth embodiment of the present invention, in combination with the above-mentioned image data expansion method, the embodiment of the present invention provides the following technical solution, a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned image data expansion method is implemented.

[0112] Those skilled in the art will appreciate that the logic and / or steps described in the flowchart as data or in other ways herein, for example, can be considered as a sequenced data table of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses.

[0113] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0114] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0115] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for image data expansion, characterized in that: The method comprises: Acquire an initial image, input the initial image into an estimation network to obtain an estimated value of the initial image, input the estimated value and the initial image into a denoising network to obtain a denoised image; transforming the denoised image to obtain the hue, saturation, and brightness of the denoised image, enhancing the brightness using a correction function and an image processing filter function, merging the processed brightness with the hue and the saturation to obtain a merged color space, and reversely transforming the color space into a brightened image; A fused image of the brightened image is extracted using a single-level feature pyramid structure, local features of the fused image are extracted using an encoder that introduces a deep convolutional layer, the local features are aggregated to form a local fused image, and a decoder is used to decode the local fused image to obtain a target image; Segment the target of the target image using the segmentation model, obtain the mask of the target, and paste, mirror, and scale the mask on the image to be expanded based on the mask using the image processing library to obtain an expanded image; The estimation network includes five convolutional neural network frameworks and a moving window mechanism; The denoising network includes two residual structures, two convolutional neural network frameworks and three multi-scale structures, wherein the multi-scale structure includes a branch unit and a fusion unit; The step of extracting the fused image of the brightened image by using a single-level feature pyramid structure comprises: The downsampling operation in the residual network is replaced by a dilated convolution group with different dilation coefficients, and multiple groups of parallel transposed convolution operations are used on its last layer output to form a single-level feature pyramid structure. Extracting a feature image containing shallow and deep feature information of the brightened image by using the single-level feature pyramid structure, and performing transposed convolution operations of different sizes on the feature image to form a fused image; The convolutional neural network framework includes a 3x3 convolution and a ReLU, and the moving window mechanism includes two 1x1 convolutions and a swin-conv structure, wherein the swin-conv structure is a Swin Transformer Convolution, and the ReLU is a Rectified Linear Unit.

2. The image data expansion method according to claim 1, characterized in that: The step of processing the brightness using a correction function and an image processing filter function comprises: Performing image processing on the brightness to obtain a first processing layer and a second processing layer; Extracting the illumination component of the first processing layer by using an image processing filter function, and calculating based on the illumination component by using a weighted calculation formula to obtain a reflection layer; The second processing layer is searched by the algorithm to obtain a correction value, and a correction function is used to correct the brightness component of each pixel point of the second processing layer based on the correction value to form a brightness layer; The brightness layer is superimposed on the reflection layer, and fusion compensation is performed to obtain the illumination layer.

3. The image data expansion method according to claim 2, characterized in that: The weighted calculation formula includes: Represented as the reflection layer, represents the brightness of the ith channel at position (x,y), Represented as a convolution operation, Expressed as an image processing filter function, Expressed as weight coefficient, The scale of the convolution kernel, Respectively represent the horizontal and vertical coordinates of the pixel; The correction function comprises: in, represents the output image, Expressed as the correction value, It is represented as pixel gray value, and q is represented as correction parameter.

4. The image data expansion method according to claim 1, characterized in that: The encoder that introduces the deep convolution layer includes: an input embedding layer, a multi-head attention layer, a residual normalization layer, a deep convolution layer and a fully connected layer, wherein the deep convolution layer is used to enhance the relevance of local information of the context.

5. The image data expansion method according to claim 1, characterized in that: A weight-constrained attention mechanism is added to the decoder; The attention mechanism includes: in, Indicates x The first x The output of the layer, , and Respectively represent x The query of each header, the corresponding value and key, represents the bias term of the residual connection, x is the index value of the layer number, T represents transposition, Represents the output of the previous layer.

6. An image data expansion system, characterized in that: The system comprises: A denoising module is used to obtain an initial image, input the initial image into an estimation network to obtain an estimated value of the initial image, and input the estimated value and the initial image into a denoising network to obtain a denoised image; a brightening module, for transforming the denoised image to obtain the hue, saturation, and brightness of the denoised image, enhancing the brightness using a correction function and an image processing filter function, merging the processed brightness with the hue and the saturation to obtain a merged color space, and reversely transforming the color space into a brightened image; A decoding module, used for extracting a fused image of the brightened image using a single-level feature pyramid structure, extracting local features of the fused image using an encoder that introduces a deep convolutional layer, aggregating the local features to form a local fused image, and decoding the local fused image using a decoder to obtain a target image; An expansion module, used to segment the target of the target image using the segmentation model, obtain the mask of the target, and use the image processing library to paste, mirror, and scale the mask on the image to be expanded based on the mask to obtain an expanded image; The estimation network includes five convolutional neural network frameworks and a moving window mechanism; The denoising network includes two residual structures, two convolutional neural network frameworks and three multi-scale structures, wherein the multi-scale structure includes a branch unit and a fusion unit; The step of extracting the fused image of the brightened image by using a single-level feature pyramid structure comprises: The downsampling operation in the residual network is replaced by a dilated convolution group with different dilation coefficients, and multiple groups of parallel transposed convolution operations are used on its last layer output to form a single-level feature pyramid structure. Extracting a feature image containing shallow and deep feature information of the brightened image by using the single-level feature pyramid structure, and performing transposed convolution operations of different sizes on the feature image to form a fused image; The convolutional neural network framework includes a 3x3 convolution and a ReLU, and the moving window mechanism includes two 1x1 convolutions and a swin-conv structure, wherein the swin-conv structure is a Swin Transformer Convolution, and the ReLU is a Rectified Linear Unit.

7. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image data expansion method according to any one of claims 1 to 5 is implemented.

8. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the image data expansion method according to any one of claims 1 to 5 is implemented.

Images