Image processing method, apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to an image processing method, device, equipment and medium, wherein the method comprises: acquiring a first image; acquiring a target parameter; wherein the target parameter is an adjustable parameter related to a target resolution of a second image to be generated; generating one-dimensional feature information corresponding to the first image through an encoder in an image tokenizer, and obtaining a one-dimensional feature unit sequence based on the target parameter and the one-dimensional feature information; and generating the second image through a decoder in the image tokenizer based on the one-dimensional feature unit sequence. Embodiments of the present disclosure can make the resolution of the generated image change accordingly with the adjustment of the target parameter, and can better improve the flexibility of image processing, and meet the diversified demand of variable image resolution.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and medium. Background Technology

[0002] Image segmenters are commonly used models in the field of image processing. They can convert information from an input image into feature units (tokens), which can then be used for image reconstruction. The inventors have found that the image segmenters used in related technologies do not perform well in image processing and have significant limitations, thus requiring further improvement. Summary of the Invention

[0003] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides an image processing method, apparatus, device and medium.

[0004] This disclosure provides an image processing method, the method comprising: acquiring a first image; acquiring target parameters; wherein the target parameters are adjustable parameters related to the target resolution of a second image to be generated; generating one-dimensional feature information corresponding to the first image through an encoder in an image segmenter, and obtaining a one-dimensional feature unit sequence based on the target parameters and the one-dimensional feature information; and generating the second image through a decoder in the image segmenter based on the one-dimensional feature unit sequence.

[0005] Optionally, obtaining a one-dimensional feature unit sequence based on the target parameters and the one-dimensional feature information includes: determining the target quantity based on the target parameters; wherein the target quantity is the number of feature units in the one-dimensional feature unit sequence used to generate the second image, and the target quantity is positively correlated with the target resolution; and obtaining a one-dimensional feature unit sequence based on the target quantity and the one-dimensional feature information.

[0006] Optionally, the target parameter includes the target quantity; or, the target parameter includes the target resolution, and the step of determining the target quantity based on the target parameter includes: obtaining a preset mapping function; and obtaining the target quantity corresponding to the target resolution based on the mapping function; wherein, the mapping function is a positive correlation function between the resolution of the image to be generated and the number of feature units in the one-dimensional feature unit sequence used to generate the image to be generated.

[0007] Optionally, the target resolution is not higher than a specified maximum resolution, which is less than or equal to the resolution of the first image; the number of targets is not higher than a specified maximum number, and the specified maximum number is determined based on the maximum number of feature units that the image segmenter can process.

[0008] Optionally, when the number of targets is greater than 1, the i-th feature unit in the one-dimensional feature unit sequence carries more image detail information than the (i-1)-th feature unit, and the i-th feature unit carries less image structure information than the (i-1)-th feature unit; where i is a value greater than or equal to 2 and less than or equal to the number of targets.

[0009] Optionally, generating the second image based on the one-dimensional feature unit sequence using the decoder in the image segmenter includes: determining the required number of mask feature units according to the target number; wherein the number of mask feature units is positively correlated with the target number; obtaining a mask feature unit sequence based on the number of mask feature units; and generating the second image based on the mask feature unit sequence and the one-dimensional feature unit sequence using the decoder in the image segmenter.

[0010] Optionally, generating the second image based on the mask feature unit sequence and the one-dimensional feature unit sequence via the decoder in the image segmenter includes: determining the two-dimensional position encoding size corresponding to the mask feature unit based on the number of mask feature units in the mask feature unit sequence; and generating the second image based on the two-dimensional position encoding size corresponding to the mask feature unit, the mask feature unit sequence, and the one-dimensional feature unit sequence via the decoder in the image segmenter.

[0011] Optionally, the one-dimensional feature information includes continuous one-dimensional implicit features; when the image segmenter further includes a quantizer, obtaining a one-dimensional feature unit sequence based on the target parameter and the one-dimensional feature information includes: converting the continuous one-dimensional implicit features into discrete one-dimensional implicit features through the quantizer; and obtaining a one-dimensional feature unit sequence based on the target parameter and the discrete one-dimensional implicit features.

[0012] Optionally, the image segmenter is trained through the following steps: acquiring a sample image; acquiring sample parameters, and acquiring a theoretical reconstructed image corresponding to the sample image based on the sample parameters; wherein the resolution of the theoretical reconstructed image is determined based on the sample parameters; generating an actual reconstructed image corresponding to the sample image using an initial segmenter based on the sample image and the sample parameters; determining a target loss value between the theoretical reconstructed image and the actual reconstructed image using a preset target loss function, training the initial segmenter based on the target loss value, and using the trained initial segmenter as the image segmenter.

[0013] This disclosure also provides an image processing apparatus, comprising: an image acquisition module for acquiring a first image; a quantity acquisition module for acquiring target parameters; wherein the target parameters are adjustable parameters related to the target resolution of a second image to be generated; an image encoding module for generating one-dimensional feature information corresponding to the first image through an encoder in an image segmenter, and obtaining a one-dimensional feature unit sequence based on the target parameters and the one-dimensional feature information; and an image decoding module for generating the second image based on the one-dimensional feature unit sequence through a decoder in the image segmenter.

[0014] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the image processing method provided in this disclosure.

[0015] This disclosure also provides a computer-readable storage medium storing a computer program for performing the image processing method provided in this disclosure.

[0016] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the image processing method provided in this disclosure.

[0017] The technical solution provided in this disclosure fully considers that existing image segmenters only support a fixed and uniform resolution, which cannot be adjusted according to needs and has significant limitations. Therefore, this disclosure provides an additional target parameter, which is an adjustable parameter related to the target resolution of the second image to be generated. This facilitates meeting the requirement of variable resolution in the generated image. The one-dimensional feature unit sequence obtained by the image segmenter is not only related to the one-dimensional feature information corresponding to the first image but also to the target parameter, thereby helping to obtain a second image with the required target resolution. Through this method, the image segmenter can adjust the one-dimensional feature unit sequence used to generate the second image based on the adjustable parameter related to the target resolution, thus helping to generate an image with the required resolution. The resolution of the generated image changes accordingly with the adjustment of the target parameter. Compared to related technologies that only support one resolution, this significantly improves the flexibility of image processing and meets the diverse needs of variable image resolution.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A schematic flowchart of an image processing method provided in an embodiment of this disclosure;

[0022] Figure 2 A schematic flowchart of an image processing method provided in an embodiment of this disclosure;

[0023] Figure 3 This is a schematic diagram of the structure of an image word segmenter provided in an embodiment of the present disclosure;

[0024] Figure 4 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present disclosure;

[0025] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0026] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0027] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0028] Figure 1 This is a flowchart illustrating an image processing method provided in an embodiment of the present disclosure. The method can be executed by an image processing device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S108:

[0029] Step S102: Obtain the first image. This embodiment of the disclosure does not limit the content of the first image. For example, to ensure the effectiveness of subsequent image processing, the first image can be a high-resolution image.

[0030] Step S104: Obtain target parameters; wherein, the target parameters are adjustable parameters related to the target resolution of the second image to be generated. This embodiment of the disclosure does not limit the target parameters; any parameter related to the target resolution is acceptable. For example, the target parameters can be directly the target resolution of the second image to be generated, or they can be intermediate parameters affecting the target resolution of the second image to be generated. For instance, it can be the number of feature units in the one-dimensional feature unit sequence used to generate the second image (which can be called the target number). The more feature units in the one-dimensional feature unit sequence, the more helpful it is in reconstructing a detailed image; that is, the higher the target resolution, the more positively correlated the target number is with the target resolution.

[0031] It should be noted that the image segmenters in the relevant technologies only support a fixed resolution, and the intermediate parameters such as the number of feature units used to generate the image to be generated are also fixed. Therefore, the above-mentioned step of obtaining adjustable target parameters does not exist.

[0032] Step S106: Generate one-dimensional feature information corresponding to the first image through the encoder in the image segmenter, and obtain a one-dimensional feature unit sequence based on the target parameters and the one-dimensional feature information.

[0033] One-dimensional feature information can be obtained based on deep features obtained after feature encoding processing of the first image by the encoder. For example, one-dimensional feature information includes continuous one-dimensional implicit features. Through the above method, a two-dimensional image can be converted into continuous one-dimensional implicit features, which helps to achieve one-dimensional word segmentation subsequently. Compared with two-dimensional word segmentation, it can effectively reduce information redundancy. Based on this, a one-dimensional feature unit sequence can be obtained based on the target parameters and one-dimensional feature information. The one-dimensional feature unit sequence can be called a one-dimensional implicit feature unit sequence. The feature units (tokens) in the one-dimensional feature unit sequence can be implicit feature units (implicit tokens). The feature unit mentioned in this embodiment is a structured and processable unit representation that carries image information and can also be understood as an intermediate representation of the image, which can be used by the model to efficiently and reliably process image information. In related technologies, the number of feature units in the one-dimensional feature unit sequence is fixed. However, this embodiment can dynamically determine the number of feature units in the one-dimensional feature unit sequence based on the obtained adjustable target parameters, thereby affecting the resolution of the subsequently generated image based on this number.

[0034] In practical applications, if the image segmenter is a discrete image segmenter, it is necessary to quantize the continuous one-dimensional feature information. That is, if the image segmenter also includes a quantizer, a one-dimensional feature unit sequence is obtained based on the target parameters and one-dimensional feature information. This includes: converting the continuous one-dimensional implicit features into discrete one-dimensional implicit features using the quantizer; and obtaining the one-dimensional feature unit sequence based on the target parameters and discrete one-dimensional implicit features. These quantization steps help ensure the controllability of the output results. However, if the image segmenter is a continuous image segmenter, the above step of quantizing the one-dimensional implicit features is unnecessary. The specific settings can be flexibly configured according to requirements.

[0035] Step S108: Based on the one-dimensional feature unit sequence, a second image is generated by the decoder in the image segmenter. Specifically, the decoder can decode based on the one-dimensional feature unit sequence to reconstruct the second image. Since the number of feature units (such as implicit tokens) in the one-dimensional feature unit sequence is determined based on adjustable target parameters, the more feature units there are, the richer the details of the reconstructed second image; the fewer feature units there are, the less details the reconstructed second image will have. Based on this, the resolution of the generated image can be adjusted flexibly and conveniently.

[0036] The technical solution provided in this disclosure fully considers that existing image segmenters only support a fixed and uniform resolution, which cannot be adjusted according to needs and has significant limitations. Therefore, this disclosure provides an additional target parameter, which is an adjustable parameter related to the target resolution of the second image to be generated. This facilitates meeting the requirement of variable resolution in the generated image. The one-dimensional feature unit sequence obtained by the image segmenter is not only related to the one-dimensional feature information corresponding to the first image but also to the target parameter, thereby helping to obtain a second image with the required target resolution. Through this method, the image segmenter can adjust the one-dimensional feature unit sequence used to generate the second image based on the adjustable parameter related to the target resolution, thus helping to generate an image with the required resolution. The resolution of the generated image changes accordingly with the adjustment of the target parameter. Compared to related technologies that only support one resolution, this significantly improves the flexibility of image processing and meets the diverse needs of variable image resolution.

[0037] In some implementations, the steps described above for obtaining a sequence of one-dimensional feature units based on target parameters and one-dimensional feature information can be performed with reference to steps a and b below:

[0038] Step a, determine the target quantity based on the target parameters; wherein, the target quantity is the number of feature units in the one-dimensional feature unit sequence used to generate the second image, and the target quantity is positively correlated with the target resolution.

[0039] As mentioned earlier, the target parameter is an adjustable parameter that can be flexibly set according to needs. The target parameter may include the target quantity, which directly determines the target number of feature units in the one-dimensional feature unit sequence used to generate the second image. Alternatively, the target parameter may include the target resolution, from which the target quantity can be deduced. In practical applications, the specific type of the target parameter can be flexibly set. For example, in scenarios targeting ordinary users, the target parameter can be set to a resolution that is easier for users to understand. In model training scenarios targeting professionals, the target parameter can be set to an intuitive target quantity. The specific setting is flexible, and the target quantity can be effectively determined through the above methods, thereby ensuring that the second image with the expected resolution can be generated. In addition, the embodiments of this disclosure fully consider the limited processing capabilities of the image segmenter, and therefore limit the range of the target parameter. For example, the target resolution is not higher than the specified maximum resolution, the maximum resolution is less than or equal to the resolution of the first image, and the target quantity is not higher than the specified maximum quantity. The specified maximum quantity is determined based on the maximum number of feature units that the image segmenter can process. For example, the specified maximum quantity is equal to the maximum number of feature units in the one-dimensional feature unit sequence that the image segmenter can process, specifically, the maximum number of implicit tokens that the image segmenter can process. The above settings can reasonably limit the adjustment range of the target parameters, thereby effectively ensuring that the image segmenter can reliably process images based on the target parameters.

[0040] When the target parameters include the target resolution, the step of determining the number of targets based on the target parameters includes: obtaining a preset mapping function; and obtaining the number of targets corresponding to the target resolution based on the mapping function. The mapping function is a positive correlation function between the resolution of the image to be generated and the number of feature units in the one-dimensional feature unit sequence used to generate the image. This method allows for efficient and quick determination of the number of targets required to meet the target resolution. For ease of understanding, the mapping function is illustrated below:

[0041] A mapping function can be used to clearly and objectively present the correlation between the resolution of the image to be generated and the number of feature units in the one-dimensional feature unit sequence used to generate the image. This mapping function can be characterized by a positive correlation function. This disclosure does not limit the positive correlation function; in some specific examples, the mapping function... It can be:

[0042]

[0043] Where R is the aforementioned maximum resolution, i.e., the currently supported maximum resolution; N is the aforementioned maximum number, i.e., the maximum number of feature units in the one-dimensional feature unit sequence that the image segmenter can process, or the maximum number of feature units that can be contained in the one-dimensional feature unit sequence used to generate the image; p is a hyperparameter, the specific value of which can be determined according to the requirements, for example, p = 1.5. As shown in the above formula, b is a fixed value obtained based on R, N, and p; n is any integer in [1, N], i.e., the aforementioned target number, which can be specified according to the requirements, or set randomly, or n can be derived from the specified resolution. In the above formula, it is explicitly stated that when n = 1, the resolution is 1, and when n = N, the resolution is R, i.e., when the one-dimensional feature unit sequence contains the maximum number of feature units, the generated image can achieve the maximum resolution, while when the one-dimensional feature unit sequence contains the minimum number of feature units (i.e., 1), the resolution of the generated image is also the minimum.

[0044] Based on the above mapping function, the resolution r of the image to be generated can be accurately determined. n For example, resolution r n The relationship between the width w and height h of the image to be generated can be referenced as follows:

[0045]

[0046] That is, resolution r n That is equal to Given the known resolution, the specific values ​​of the height h and width w of the second image to be generated can be determined accordingly. The above is just an example; other relationships between image resolution and image width and height can be set as needed, which are not limited here.

[0047] Step b: Obtain a sequence of one-dimensional feature units based on the number of targets and one-dimensional feature information.

[0048] In some implementations, when the target number is greater than 1, the i-th feature unit in the one-dimensional feature unit sequence carries more image detail information than the (i-1)-th feature unit, and the i-th feature unit carries less image structure information than the (i-1)-th feature unit; where i is a value greater than or equal to 2 and less than or equal to the target number. That is, the embodiments of this disclosure assign a reasonable order to the feature units in the one-dimensional feature unit sequence used to generate images. Specifically, the feature units in the one-dimensional feature unit sequence are arranged sequentially in order of increasing detail information and decreasing structure information. Compared to the related technologies that use a fixed number of feature units with disorganized information, the above method is more consistent with the human perception process of images, contributing to high-quality image generation. It also facilitates subsequent network learning scenarios that require image reconstruction based on the one-dimensional feature unit sequence, improving the training efficiency and image quality of models such as autoregressive image generation models.

[0049] In some examples, multiple implicit feature units can be generated based on one-dimensional feature information, following an order of increasing detail information and decreasing structural information. The number of generated implicit feature units can be a specified maximum. Then, a target number of implicit feature units are selected from these multiple implicit feature units in a sequential order to form the desired one-dimensional feature unit sequence. Alternatively, one can directly generate only the target number of implicit feature units based on the one-dimensional feature information. For example, the first implicit feature unit, which carries the most image structural information but the least image detail information, can be generated first. Then, subsequent implicit feature units are generated in a similar manner, following the rule of increasing detail information and decreasing structural information, until the number of generated implicit feature units equals the target number. The desired one-dimensional feature unit sequence is then formed based on the target number of implicit feature units. This approach yields a one-dimensional feature unit sequence with a reasonable information distribution and a feature unit number related to the target resolution. This not only facilitates the generation of high-quality images but also ensures that the resolution of the generated images meets the requirements.

[0050] In some implementations, the step of generating a second image based on a one-dimensional feature unit sequence using a decoder in an image segmenter can be performed with reference to steps 1 to 3 below:

[0051] Step 1: Determine the required number of mask feature units based on the number of targets; wherein the number of mask feature units is positively correlated with the number of targets. In practical applications, the decoder needs to use mask feature units (mask tokens) to perform decoding. The number of mask feature units used in related technologies is fixed, while the embodiments of this disclosure can determine the number of mask feature units based on the number of targets related to resolution. The more targets there are in the one-dimensional feature unit sequence, the more mask feature units are required, thereby helping to generate higher resolution images with richer details.

[0052] Step 2: Obtain the mask feature unit sequence based on the number of mask feature units. Initially, the mask feature units may not carry any information; subsequently, the information carried by the one-dimensional feature unit sequence can be gradually integrated into the mask feature units in the decoder to facilitate subsequent image reconstruction. In this embodiment, an initial mask feature unit sequence can be constructed first based on the number of mask feature units.

[0053] Step 3: Based on the mask feature unit sequence and the one-dimensional feature unit sequence, generate the second image through the decoder in the image segmenter. The mask feature unit sequence and the one-dimensional feature unit sequence can be input into the decoder in the image segmenter. The decoding process of the decoder can be referred to relevant technologies and will not be elaborated here. Then, the output of the decoder can be mapped back to the pixel space through a fully connected layer to obtain the second image. Since the number of mask feature units in the mask feature unit sequence is not fixed but dynamically adjusted based on the resolution of the desired generated image, in some specific implementations, step 3 can be performed as follows: Steps 3.1 and 3.2.

[0054] Step 3.1: Determine the two-dimensional position coding size corresponding to the mask feature unit based on the number of mask feature units in the mask feature unit sequence. In related technologies, the number of mask feature units, the two-dimensional position coding size, and the resolution of the generated image are all fixed. However, the embodiments of this disclosure can dynamically adjust the number of mask feature units and dynamically adjust the two-dimensional position coding size corresponding to the mask feature units based on this number, so that the image of the required resolution can be reconstructed subsequently.

[0055] Step 3.2: Based on the two-dimensional positional encoding size, mask feature unit sequence, and one-dimensional feature unit sequence corresponding to the mask feature units, the second image is generated by the decoder in the image segmenter. The specific method for reconstructing the second image using the decoder, based on the known two-dimensional positional encoding size, mask feature unit sequence, and one-dimensional feature unit sequence corresponding to the mask feature units, can be found in relevant technologies and will not be elaborated here.

[0056] By using the above methods, it is possible to ensure that the resolution of the generated image meets expectations, and the resolution of the generated image can be changed accordingly as the target parameters are adjusted, which can improve the flexibility of image processing and meet the diverse needs of variable image resolution.

[0057] Based on the foregoing, see also Figure 2 The flowchart of an image processing method shown mainly includes the following steps S202 to S214:

[0058] Step S202: Obtain the first image.

[0059] Step S204: Obtain target parameters; wherein, the target parameters are adjustable parameters related to the target resolution of the second image to be generated.

[0060] Step S206: Generate one-dimensional feature information corresponding to the first image through the encoder in the image segmenter, wherein the one-dimensional feature information includes continuous one-dimensional implicit features.

[0061] Step S208: Determine the number of targets based on the target parameters, and obtain a one-dimensional feature unit sequence based on the number of targets and one-dimensional feature information. The number of targets is the number of feature units in the one-dimensional feature unit sequence used to generate the second image, and the number of targets is positively correlated with the target resolution. The feature units in the one-dimensional feature unit sequence are implicit feature units.

[0062] Step S210: Determine the required number of mask feature units based on the target quantity.

[0063] Step S212: Obtain the mask feature unit sequence based on the number of mask feature units.

[0064] Step S214: Based on the mask feature unit sequence and the one-dimensional feature unit sequence, a second image is generated by the decoder in the image segmenter.

[0065] The specific implementation of the above steps can be found in the aforementioned content. This method allows for the acquisition of customizable target parameters, which are adjustable parameters related to the target resolution of the second image to be generated. This effectively meets the requirement of variable resolution in the generated image. The number of target parameters is positively correlated with the target resolution. Based on the target parameters, the number of feature units in the one-dimensional feature unit sequence used to generate the second image can be obtained. Based on this, an image segmenter can obtain the one-dimensional feature unit sequence used to generate the image, thus obtaining a second image with the required target resolution. In this method, the image segmenter can adjust the number of generated feature units according to the user-set parameters related to the target resolution, thereby generating an image that meets the user's resolution requirements, offering high flexibility.

[0066] For easier understanding, you can also refer to, for example Figure 3 The diagram shown is a structural schematic of an image word segmenter. Figure 3 The image segmenter in the example mainly consists of an encoder, a quantizer, and a decoder, which are explained below:

[0067] This disclosure does not limit the specific implementation of the encoder. For example, the encoder can be implemented based on a ViT (Vision Transformer) structure. In practical applications, the first image can first be divided into multiple non-overlapping image blocks. Then, a fully connected layer maps these image blocks into a sequence of image feature units (i.e., image tokens). The image feature unit sequence and a one-dimensional query feature unit sequence (i.e., query tokens, specifically...) are then... Figure 3 q1~q N The images are concatenated together and input into the encoder. The encoder can process information between image feature units based on a bidirectional attention mechanism, and process information between query feature units based on a unidirectional causal attention mechanism. It can also provide information from image feature units to query feature units, that is, integrate the information of feature units, thereby embedding all the image information of the first image into a one-dimensional feature unit sequence. The output result corresponding to the query feature unit sequence is the continuous one-dimensional implicit image representation information (specifically...). Figure 3 C1 to C N Specifically, this can also be referred to as the implicit representation feature of a one-dimensional image, that is, the one-dimensional feature information corresponding to the aforementioned first image.

[0068] The quantizer is a network module with a codebook, containing several codebook vectors. The quantizer calculates the distance between the input continuous features and each codebook vector, selecting the closest codebook vector as the quantized feature. Through the quantizer, the continuous one-dimensional image implicit representation information can be quantized into discrete one-dimensional image implicit representation information. It should be noted that the quantizer in the image segmenter is a configurable module, and its configuration can be tailored to specific needs. For example, if a discrete model is required, a quantizer can be configured in the image segmenter; if a continuous model is required, a quantizer is not necessary.

[0069] This disclosure does not limit the specific implementation of the decoder. For example, the decoder can also be implemented based on a ViT (Vision Transformer) structure. In practical applications, it can be assumed that the quantizer outputs N implicit feature units. This disclosure only selects the first n (i.e., the aforementioned target number) implicit feature units (i.e., implicit tokens, specifically...). Figure 3 Z1~Z NThis constitutes the implicit feature unit sequence to be processed (i.e., the aforementioned one-dimensional feature unit sequence), and all other implicit feature units thereafter can be lost (in...). Figure 3 (Illustrated by an ×). In practical applications, the required number of mask feature units (i.e., mask tokens) can be determined based on the target quantity, thus obtaining a mask feature unit sequence. Subsequently, the implicit feature unit sequence and the mask feature unit sequence can be input into the decoder. The decoder performs decoding based on the implicit feature unit sequence and the mask feature unit sequence. As mentioned earlier, the implicit feature units in the implicit feature unit sequence are arranged in order of increasing detail information and decreasing structural information. This method is more in line with the human perception process of images. When reconstructing an image, the decoder can focus on the overall structure of the image based on the earlier implicit feature units and supplement the image details based on the later implicit feature units. It can be understood that as the number of implicit feature units in the implicit feature unit sequence increases, detailed information can be gradually supplemented, thereby helping to improve the resolution of the reconstructed image.

[0070] In practical applications, the input feature units of the encoder and decoder are typically coupled with corresponding positional encodings. Specifically, the image feature unit sequence corresponding to the encoder and the mask feature unit sequence corresponding to the decoder can be coupled with two-dimensional positional encodings to effectively establish a correlation with the corresponding positions of the feature units on the image. For the query feature unit sequence corresponding to the encoder and the implicit feature unit sequence corresponding to the decoder, one-dimensional positional encodings are sufficient. Furthermore, by adjusting the size of the two-dimensional positional encodings corresponding to the mask feature unit sequences, the variable resolution encoding and decoding of the image segmenter can be better adapted, ultimately reconstructing an image that meets the required resolution.

[0071] By employing the above method, a mapping relationship between the number of feature units and the reconstruction resolution can be established. This allows the one-dimensional image segmenter to dynamically decode images at the required resolution based on an adjustable number of feature units, thus achieving multi-resolution decoding and moving beyond the limitation of only supporting the decoding and generation of images at one resolution. Furthermore, the information distribution pattern of the aforementioned feature units (i.e., decreasing structural information and increasing detail information) facilitates model learning and processing, effectively ensuring the quality of image generation.

[0072] Furthermore, this disclosure provides a method for obtaining an image word segmenter. Exemplarily, the image word segmenter is obtained through training via the following steps A to D:

[0073] Step A: Obtain the sample image.

[0074] Step B involves obtaining sample parameters and, based on these parameters, acquiring the theoretically reconstructed image corresponding to the sample image. The resolution of the theoretically reconstructed image is determined based on the sample parameters. The theoretically reconstructed image is the image expected to meet the requirements, generated by the image segmenter. In practical applications, the resolution of the theoretically reconstructed image can be determined first based on the sample parameters, and then the theoretically reconstructed image with that resolution can be acquired.

[0075] Step C: Based on the sample image and sample parameters, generate the actual reconstructed image corresponding to the sample image using the initial word segmenter. The method for generating the actual reconstructed image can refer to the method for generating the second image mentioned above, and will not be repeated here.

[0076] Step D involves determining the target loss value between the theoretically reconstructed image and the actual reconstructed image using a preset target loss function. An initial word segmenter is then trained based on this target loss value, and the trained initial word segmenter is used as the image word segmenter. This embodiment does not limit the target loss function. Exemplarily, the target loss function includes one or more of reconstruction loss, perceptual loss, and GAN (Generative Adversarial Networks) loss. This target loss function can objectively and accurately measure the difference between the theoretically reconstructed image and the actual reconstructed image, and train the initial word segmenter in the direction of reducing the target loss value, thereby improving the image reconstruction capability of the initial word segmenter and enabling it to reconstruct images that meet expectations. Training ends when the target loss value converges to a preset threshold range, resulting in the desired image word segmenter. Using a theoretically reconstructed image with the required resolution, the word segmenter can be supervised during training to perform image reconstruction based on a specified number of feature units, thereby controlling the information distribution of the feature units. The trained image word segmenter can generate images that meet expectations based on dynamically set adjustable parameters.

[0077] In summary, the embodiments of this disclosure fully consider that the method of embedding image information into two-dimensional feature units in related technologies has a lot of spatial redundancy and requires a large number of feature units. Therefore, compressing the image information of the first image into a one-dimensional feature unit sequence can effectively reduce the number of feature units required and improve the spatial redundancy problem existing in the prior art. Moreover, the embodiments of this disclosure also support dynamically adjustable target parameters related to the resolution of the image to be generated. Dynamically adjusting the number of feature units in the one-dimensional feature unit sequence based on the target parameters not only helps to obtain a reconstructed image with the required target resolution, but also gives the feature units in the one-dimensional feature unit sequence a reasonable order. These feature units are arranged in order of increasing detail information and decreasing structural information. This method is more in line with the human perception process of images, which helps to generate high-quality images. It is also easier for subsequent network learning that needs to perform image reconstruction based on the one-dimensional feature unit sequence, improving the training efficiency and image quality of models such as autoregressive image generation models.

[0078] Corresponding to the aforementioned image processing method, this disclosure further provides an image processing apparatus. Figure 4 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present disclosure. The apparatus can be implemented by software and / or hardware, and is generally integrated into an electronic device, such as... Figure 4 As shown, the image processing apparatus includes:

[0079] Image acquisition module 402 is used to acquire the first image;

[0080] The quantity acquisition module 404 is used to acquire target parameters; wherein, the target parameters are adjustable parameters related to the target resolution of the second image to be generated;

[0081] The image encoding module 406 is used to generate one-dimensional feature information corresponding to the first image through the encoder in the image segmenter, and to obtain a one-dimensional feature unit sequence based on the target parameters and the one-dimensional feature information.

[0082] Image decoding module 408 is used to generate a second image based on a sequence of one-dimensional feature units through a decoder in an image segmenter.

[0083] With the above-mentioned device, the image segmenter can adjust the sequence of one-dimensional feature units used to generate the second image based on the adjustable parameters related to the target resolution, thereby helping to generate an image with the required resolution. The resolution of the generated image will change accordingly with the adjustment of the target parameters. Compared with related technologies that only support one resolution, it can better improve the flexibility of image processing and meet the diverse needs of variable image resolution.

[0084] In some embodiments, the image encoding module 406 is specifically used to: determine the target quantity based on the target parameters; wherein the target quantity is the number of feature units in the one-dimensional feature unit sequence used to generate the second image, and the target quantity is positively correlated with the target resolution; and obtain a one-dimensional feature unit sequence based on the target quantity and the one-dimensional feature information.

[0085] In some implementations, the target parameter includes the target quantity; or, the target parameter includes the target resolution, and the image encoding module 406 is specifically used to: obtain a preset mapping function; and based on the mapping function, obtain the target quantity corresponding to the target resolution; wherein, the mapping function is a positive correlation function between the resolution of the image to be generated and the number of feature units in the one-dimensional feature unit sequence used to generate the image to be generated.

[0086] In some implementations, the target resolution is not higher than a specified maximum resolution, which is less than or equal to the resolution of the first image; the target number is not higher than a specified maximum number, and the specified maximum number is determined based on the maximum number of feature units that the image segmenter can process.

[0087] In some implementations, when the number of targets is greater than 1, the i-th feature unit in the one-dimensional feature unit sequence carries more image detail information than the (i-1)-th feature unit, and the i-th feature unit carries less image structure information than the (i-1)-th feature unit; where i is a value greater than or equal to 2 and less than or equal to the number of targets.

[0088] In some embodiments, the image decoding module 408 is specifically used to: determine the required number of mask feature units based on the target number; wherein the number of mask feature units is positively correlated with the target number; obtain a mask feature unit sequence based on the number of mask feature units; and generate the second image through the decoder in the image segmenter based on the mask feature unit sequence and the one-dimensional feature unit sequence.

[0089] In some embodiments, the image decoding module 408 is specifically used to: determine the two-dimensional position encoding size corresponding to the mask feature unit based on the number of mask feature units in the mask feature unit sequence; and generate the second image through the decoder in the image segmenter based on the two-dimensional position encoding size corresponding to the mask feature unit, the mask feature unit sequence, and the one-dimensional feature unit sequence.

[0090] In some implementations, the one-dimensional feature information comprises continuous one-dimensional implicit features;

[0091] In some embodiments, where the image segmenter further includes a quantizer, the image encoding module 406 is specifically used to: convert the continuous one-dimensional implicit features into discrete one-dimensional implicit features through the quantizer; and obtain a sequence of one-dimensional feature units based on the target parameters and the discrete one-dimensional implicit features.

[0092] In some embodiments, the apparatus further includes a training module for training the image segmenter through the following steps: acquiring a sample image; acquiring sample parameters, and acquiring a theoretical reconstructed image corresponding to the sample image based on the sample parameters; wherein the resolution of the theoretical reconstructed image is determined based on the sample parameters; generating an actual reconstructed image corresponding to the sample image using an initial segmenter based on the sample image and the sample parameters; determining a target loss value between the theoretical reconstructed image and the actual reconstructed image using a preset target loss function, training the initial segmenter based on the target loss value, and using the trained initial segmenter as the image segmenter.

[0093] The image processing apparatus provided in this disclosure can execute the image processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0094] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0095] This disclosure provides an electronic device, which includes: a storage device storing a computer program thereon; and a processing device for executing the computer program in the storage device to implement the steps of any method of this disclosure.

[0096] The following is for reference. Figure 5 This document illustrates a structural schematic diagram of an electronic device 500 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Devices), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital televisions and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0097] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0098] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0099] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0100] In addition to the methods and apparatus described above, embodiments of this disclosure can also be computer program products, comprising computer program instructions that, when executed by a processor, cause the processor to perform the methods provided in the embodiments of this disclosure. The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0101] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the image processing method provided in embodiments of this disclosure.

[0102] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0103] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the image processing method of this disclosure.

[0104] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0105] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0106] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0107] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0108] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0109] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, include: Get the first image; Obtain target parameters; wherein, the target parameters are adjustable parameters related to the target resolution of the second image to be generated; The encoder in the image segmenter generates one-dimensional feature information corresponding to the first image, and a one-dimensional feature unit sequence is obtained based on the target parameters and the one-dimensional feature information. The second image is generated based on the one-dimensional feature unit sequence through the decoder in the image segmenter.

2. The method according to claim 1, characterized in that, The process of obtaining a one-dimensional feature unit sequence based on the target parameters and the one-dimensional feature information includes: The target quantity is determined based on the target parameters; wherein, the target quantity is the number of feature units in the one-dimensional feature unit sequence used to generate the second image, and the target quantity is positively correlated with the target resolution; A sequence of one-dimensional feature units is obtained based on the target quantity and the one-dimensional feature information.

3. The method according to claim 2, characterized in that, The target parameter includes the target quantity; or, The target parameters include the target resolution, and the step of determining the number of targets based on the target parameters includes: Obtain a preset mapping function; based on the mapping function, obtain the number of targets corresponding to the target resolution; wherein, the mapping function is a positive correlation function between the resolution of the image to be generated and the number of feature units in the one-dimensional feature unit sequence used to generate the image to be generated.

4. The method according to claim 2, characterized in that, The target resolution is not higher than the specified maximum resolution, and the maximum resolution is less than or equal to the resolution of the first image; The target number is no higher than the specified maximum number, and the specified maximum number is determined based on the maximum number of feature units that the image segmenter can process.

5. The method according to claim 2, characterized in that, When the number of targets is greater than 1, the i-th feature unit in the one-dimensional feature unit sequence carries more image detail information than the (i-1)-th feature unit, and the i-th feature unit carries less image structure information than the (i-1)-th feature unit; where i is a value greater than or equal to 2 and less than or equal to the number of targets.

6. The method according to claim 2, characterized in that, The step of generating the second image based on the one-dimensional feature unit sequence through the decoder in the image segmenter includes: The required number of mask feature units is determined based on the target number; wherein the number of mask feature units is positively correlated with the target number. Based on the number of mask feature units, obtain the mask feature unit sequence; The second image is generated by the decoder in the image segmenter based on the mask feature unit sequence and the one-dimensional feature unit sequence.

7. The method according to claim 6, characterized in that, The step of generating the second image based on the mask feature unit sequence and the one-dimensional feature unit sequence through the decoder in the image segmenter includes: Based on the number of mask feature units in the mask feature unit sequence, the two-dimensional position encoding size corresponding to the mask feature unit is determined; The second image is generated by the decoder in the image segmenter based on the two-dimensional positional encoding size corresponding to the mask feature unit, the mask feature unit sequence, and the one-dimensional feature unit sequence.

8. The method according to claim 1, characterized in that, The one-dimensional feature information includes continuous one-dimensional implicit features; When the image segmenter further includes a quantizer, obtaining the one-dimensional feature unit sequence based on the target parameters and the one-dimensional feature information includes: The quantizer converts the continuous one-dimensional implicit features into discrete one-dimensional implicit features. Based on the target parameters and the discrete one-dimensional implicit features, a sequence of one-dimensional feature units is obtained.

9. The method according to any one of claims 1 to 8, characterized in that, The image word segmenter was trained through the following steps: Acquire sample images; Obtain sample parameters, and obtain the theoretical reconstructed image corresponding to the sample image based on the sample parameters; wherein the resolution of the theoretical reconstructed image is determined based on the sample parameters; Based on the sample image and the sample parameters, the actual reconstructed image corresponding to the sample image is generated by the initial word segmenter; A target loss value is determined between the theoretically reconstructed image and the actual reconstructed image using a preset target loss function. The initial word segmenter is trained based on the target loss value and then used as the image word segmenter.

10. An image processing apparatus, characterized in that, include: The image acquisition module is used to acquire the first image; A quantity acquisition module is used to acquire target parameters; wherein, the target parameters are adjustable parameters related to the target resolution of the second image to be generated; The image encoding module is used to generate one-dimensional feature information corresponding to the first image through the encoder in the image segmenter, and to obtain a one-dimensional feature unit sequence based on the target parameters and the one-dimensional feature information; An image decoding module is used to generate the second image based on the one-dimensional feature unit sequence through the decoder in the image segmenter.

11. An electronic device, characterized in that, The electronic device includes: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the image processing method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the image processing method according to any one of claims 1-9.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the image processing method according to any one of claims 1-9.

Images