Personalized home appliance product design method and device based on visual autoregressive model

Abstract

The application discloses a personalized household appliance product design method and device based on a visual autoregressive model, and the method comprises the following steps: given a target reference image of a household appliance product, converting the target reference image into a series of discrete Token sequences through a visual autoregressive model, and generating a vectorized representation of the target reference image based on iterative optimization of the Token sequences; adding noise disturbance to the vectorized representation to generate a fission feature sequence and decoding to generate a fission image; generating an initial editing feature based on the vectorized representation, marking an editing area of the target reference image by using a mask matrix, and then giving an auxiliary target reference image, extracting a feature sequence of the auxiliary target reference image, and obtaining an editing feature sequence by scale-by-scale fusion based on the initial feature, the mask matrix marking and the feature sequence, and decoding to generate an editing image, thereby completing the personalized household appliance product design. The visual autoregressive model of the household appliance product is constructed, the global disturbance and the local editing are cooperatively controlled in the feature space, and the household appliance image generation effect is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and in particular relates to a personalized household appliance product design method and device based on a visual autoregressive model. BACKGROUND

[0002] In the past two years, the field of image generation has received extensive attention, especially the open source of stable diffusion models and the continuous updates of DALL-E series models, which have collectively taken the field of image generation to a new level. Although diffusion models have excellent image generation capabilities, their application in image editing and other tasks faces enormous challenges. This is mainly due to the hallucinations that may exist in synthetic images, which cannot strictly follow the editing instructions or cannot be consistent with the conditional images.

[0003] Given one or more images of concepts as input, the purpose of personalized image generation is to generate image variants of the given concept or identity. As with DreamBooth and Textual Inversion, by designing a richer inversion space, relevant concepts are extracted from a set of images, and new images are regenerated. Textual Inversion is a method of making text-to-image models better understand and generate specific concepts through backpropagation, i.e., freezing the parameters of a pre-trained model during the training process.

[0004] However, existing personalized image generation methods input reference images, text or depth maps, line drawings, encoding information, and other conditions into the generation model to generate the desired personalized image, thus presenting the following challenges: First, text-driven image editing effects are highly dependent on user description expressions. If the instructions are ambiguous, the model's results are often difficult to completely match the user's actual expectations. Second, for tasks that require fine operations, such as boundary processing, target replacement in complex occlusion scenarios, etc., existing methods are difficult to achieve accurate and detailed editing. Finally, these methods usually rely on a large amount of high-quality training data and high-performance computing resources, requiring high data quality and computing power, which poses a certain threshold for practical applications.

[0005] Autoregressive models are a class of generative models that treat text-to-image generation as a sequence-to-sequence modeling problem, similar to machine translation. Early pioneering work, such as VQ-VAE, introduced the concept of vector quantization, allowing images to be encoded as discrete token sequences, similar to the way language is processed. This approach allows the model to process visual data in a similar way to processing language tokens. However, efforts to use these models for conditional generation tasks have been relatively limited.

[0006] Other studies attempt to improve autoregressive models by reevaluating the next token prediction paradigm, challenging the traditional line-by-line raster scan approach to generating image tokens, arguing that images require more global context than text. Inspired by the masked autoencoder, MaskGIT adopts an iterative masking approach that learns to predict random masked tokens by focusing on other tokens and iteratively decoding during inference. VAR visual autoregressive model is to change the next token prediction paradigm into the next scale prediction from coarse to fine, so as to significantly improve the visual quality of generated images.

[0007] In the field of home appliance design, the current home appliance large model mainly includes natural semantic large model, visual large model, i.e. video large model, and general large model. Different from the general large model, the training data of the home appliance large model contains a large amount of mature marketing methodology and marketing data. And the existing home appliance large model is basically based on the home appliance product data set constructed by itself, and the fine-tuning pre-training large model. However, the quality and quantity of the training data set greatly affect the generation effect of these large models. At present, there is a lack of training data set for home appliance products.

[0008] Therefore, in order to solve the problems of lack of training data set and that the traditional general large model cannot realize more accurate personalized image generation in the field of home appliance design, the present application provides a method for personalized design of home appliance products based on a visual autoregressive model. SUMMARY

[0009] The present application provides a method and device for personalized design of home appliance products based on a visual autoregressive model. For the field of personalized design of home appliance products, a more accurate image personalized generation technology is provided, which is used to realize the cooperative control of global disturbance and local editing in a feature space based on a reference image of a home appliance product in the case of lacking training data, to realize the detail replacement of different home appliance products, and to improve the home appliance image generation effect of the large model.

[0010] A personalized home appliance product design method based on a visual autoregressive model, comprising the following steps:

[0011] (1) Given a target reference image of a home appliance product, convert it into a series of discrete Token sequences through a visual autoregressive model, and generate a vectorized representation of the target reference image based on the discrete Token sequences;

[0012] (2) Add noise disturbance to the vectorized representation of the target reference image to generate a fission feature sequence, and decode to generate a fission image similar to the target reference image;

[0013] (3) Generate initial features based on the vectorized representation of the target reference image, use the mask matrix to mark the area to be edited of the target reference image, and given the auxiliary target reference image of the home appliance product, extract the feature sequence of the auxiliary target reference image. Based on the initial features, the mask matrix marks and the feature sequence of the auxiliary target reference image, perform feature fusion scale by scale to obtain the editing feature sequence, decode and generate the editing image, and complete the personalized home appliance product design.

[0014] In one embodiment, step (1) specifically includes: a target reference image of a home appliance product , which is encoded by the VQ-VAE encoder in the visual autoregressive model as Token sequences with different scales of discreteness As the true label, based on the discrete Token sequence, the vectorized representation of the target reference image is generated through iterative optimization using a training-free method .

[0015] In one embodiment, when using the training-free method, the loss function is set to iteratively optimize and generate a vectorized representation of the target reference image. , the calculation formula is as follows:

[0016] ,

[0017] ,

[0018] ,

[0019] Among them, codebook represents the size of the Token codebook in the VQ-VAE encoder. Represents the discrete Token index variable in the codebook, represents the visual autoregressive model, Represents the first prediction based on the visual autoregressive model The characteristic sequence of the scale, is the category label, Represents feature sequences of different scales predicted based on the visual autoregressive model, is the number of feature maps.

[0020] In one embodiment, step (2) specifically includes: vectorizing the target reference image Inject Gaussian noise , based on the visual autoregressive model, the fission features of the next scale are predicted, and the fission features are fused scale by scale to obtain the fission feature sequences of different scales , the calculation formula is as follows:

[0021] ,

[0022] ,

[0023] ,

[0024] wherein, denotes the sequence of fissure features of the th scale, denotes the sequence of fissure features of the th scale, is the class label, is the sequence of fissure features of a different scale, is the number of fissure feature sequences;

[0025] The fissure feature sequence is input into the VQ-VAE decoder to generate a fissure image similar to the target reference image.

[0026] In one embodiment, in step (3), the using a mask matrix to mark the region to be edited comprises: randomly framing the editing region on the target reference image, generating a binary mask or a Gaussian mask, and marking the region to be edited based on the binary mask or the Gaussian mask matrix.

[0027] In one embodiment, the binary mask comprises: the editing region marked by random framing is marked as 1, indicating that the original content is retained, and the unselected editing region is marked as 0, indicating that it is replaced by new content.

[0028] In one embodiment, in step (3), the given auxiliary target reference image of a home appliance product, and extracting the feature sequence of the auxiliary target reference image comprises: the auxiliary target reference image is decomposed into a multi-scale feature sequence , denotes the discrete Token feature map of the th scale after the decomposition of the auxiliary target reference image.

[0029] In one embodiment, in step (3), the based on the initial feature, the mask matrix mark, and the feature sequence of the auxiliary target reference image, the feature fusion is performed scale by scale to obtain the editing feature sequence, comprising:

[0030] For the mask region in the initial feature, the current feature is retained;

[0031] For the non-mask region in the initial feature, the feature corresponding to the layer of the auxiliary target reference image is introduced;

[0032] The current feature Corresponding to a feature of a target reference image Fusing the next scale mask feature of the post-prediction, calculating the mask feature by scale weighting, obtaining the edited feature sequence , the calculation formula is as follows:

[0033] ,

[0034] ,

[0035] ,

[0036] Among them, indicates the vectorization of the target reference image, is a category label, is an initial edited feature obtained by the vectorization of the target reference image, indicates the edited feature sequence of the th scale, indicates the edited feature sequence of different scales predicted based on the visual autoregressive model, indicates the number of edited feature sequences, and Mask is a mask matrix.

[0037] In one embodiment, the decoding in step (3) generates an edited image, comprising: inputting the output edited feature sequence to a VQ-VAE decoder to generate an edited image.

[0038] In another aspect, the application also provides a personalized home appliance product design device based on a visual autoregressive model, comprising a memory and a processor, the memory is used to store a computer program, characterized in that the processor is used to realize the personalized home appliance product design method based on the visual autoregressive model when executing the computer program.

[0039] Compared with the prior art, the application has the following beneficial effects:

[0040] (1) By combining Textual Inversion with a visual autoregressive model, a visual autoregressive model architecture specific to home appliance products is constructed, and the vectorization is optimized in a training-free manner to adapt to the data scarcity scenario in the home appliance field.

[0041] (2) The reference image of the home appliance product is converted into an embedding feature, and the image fission and image editing are tightly coupled through feature space operations, realizing the local fusion of multiple home appliance product images and the replacement of details of different home appliance products, avoiding the instability of pixel-level operations, and realizing precise and controllable semantic-level transformation and control.

[0042] (3) Depend on pre-training model, get efficient home appliance product vectorization representation through cross-entropy optimization, can realize home appliance product style migration in the case of only a small number of example samples, reduce 98% of training data demand compared with traditional fine-tuning method, save training resources, especially in the application scene of high requirement for details such as home appliance products, can accurately replace and edit the local details of the product, ensure that the final effect of the home appliance product is consistent with the design expectation, greatly expand the application range of high-quality image editing. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 An overall schematic view of a personalized home appliance product design process based on a visual autoregressive model is provided for the embodiments of the present application.

[0044] Figure 2 A flowchart of a personalized home appliance product design method based on a visual autoregressive model is provided for the embodiments of the present application.

[0045] Figure 3 A flowchart of iterative optimization of target reference image vectorization representation is provided for the embodiments of the present application.

[0046] Figure 4 An effect diagram of personalized home appliance product image reconstruction and fission based on a visual autoregressive model is provided for the embodiments of the present application.

[0047] Figure 5 A flowchart of mask fusion based on multiple images is provided for the embodiments of the present application.

[0048] Figure 6 An effect diagram of personalized home appliance product image editing based on a visual autoregressive model is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0049] The present application will be further described in detail below in combination with the drawings and embodiments, and it should be pointed out that the following embodiments are intended to facilitate the understanding of the present application and do not have any limiting effect on the present application.

[0050] The present application aims at the problems of missing training data set and traditional general large model unable to realize more accurate personalized image generation in the field of current home appliance design, and proposes a personalized home appliance product design method based on a visual autoregressive model, as shown in Figure 1 The visual autoregressive model is used to realize home appliance product image vectorization, that is, a vector is used to restore the entire image; image fission and image editing are realized respectively on the basis of image vectorization to complete personalized home appliance product design. As shown in Figure 2 The specific implementation steps are as follows:

[0051] (1) Given a target reference image of a home appliance product, it is converted into a series of discrete Token sequences through a visual autoregressive model, and a vectorized representation of the target reference image is generated based on iterative optimization of the discrete Token sequence.

[0052] In the embodiment, a visual autoregressive model is selected as the basic model. The autoregressive model is a type of probabilistic model that decomposes complex data distribution into the product of conditional probabilities, thereby generating target data in sequence. In the visual autoregressive model, the visual image data must first be converted into a series of discrete tokens. After representing the visual image content as an ordered sequence of tokens, the autoregressive model predicts the token sequence of the next scale through the token sequences of all previous scales. This process can be expressed by the following formula:

[0053] ,

[0054] in, It means predicting the probability distribution of the current token given a historical token sequence; A discrete Token sequence representing the entire visual image, Indicates the discrete Token, i represents the index variable of the discrete Token sequence, Indicates the number of discrete Token sequence index variables, Indicates the The known Token sequence before the position, represents the parameters of the autoregressive model.

[0055] The visual autoregressive model consists of the following modules: a VQ-VAE encoder, a VQ-VAE decoder, and an autoregressive model. The VQ-VAE encoder encodes visual image data into discrete token sequences of different scales; the VQ-VAE decoder restores token sequences of different scales back to visual data; and the autoregressive model predicts the token sequence of the next scale.

[0056] Next, the user provides a reference image, and the visual autoregressive model continuously iteratively optimizes a vector, which can be reversely reconstructed to restore the entire reference image. Specifically:

[0057] like Figure 3 As shown, a target reference image of a home appliance product , which is encoded by the VQ-VAE encoder in the visual autoregressive model as Token sequences with different scales of discreteness As a real label, based on the discrete Token sequence, the vectorized representation of the target reference image is iteratively optimized by a training-free method The role of the vectorized representation is to send it into the visual autoregressive model to restore the target reference image.

[0058] The method of Textual Inversion is applied to the visual autoregressive model to construct a personalized household appliance generation model. By freezing the pre-trained model parameters, the vectorized representation is optimized in a training-free manner , which is suitable for the data scarce scenario in the household appliance field. The loss function is represented by cross-entropy, and the calculation formula is as follows:

[0059] ,

[0060] ,

[0061] ,

[0062] Wherein, codebook represents the Token codebook size in the VQ-VAE encoder, represents the discrete Token index variable in the codebook, represents the visual autoregressive model, represents the feature sequence of the first scale predicted based on the visual autoregressive model, is the category label, represents the feature sequence of different scales predicted based on the visual autoregressive model, is the number of feature maps;

[0063] By calculating the cross-entropy between the feature sequence and the real label , the vectorized representation of the target reference image is iteratively optimized .

[0064] (2) Add noise disturbance to the vectorized representation of the target reference image to generate a fission feature sequence, and decode to generate a fission image similar to the target reference image.

[0065] In the embodiment, the picture fission is to generate a similar image from a target reference image, which can be realized on the basis of the vectorized representation of the image. Specifically:

[0066] The vectorized representation of the target reference image is injected with Gaussian noise , so that from The predicted multi-scale feature sequence is no longer a simple restoration of the original image, but can obtain similar variants of the target reference image, thus achieving global fine-tuning of home appliance products, such as Figure 4 As shown, it can generate images that are similar to the reference image but also have certain differences.

[0067] Based on the visual autoregressive model VAR, the fission features of the next scale are predicted, and the fission features are fused scale by scale to obtain the fission feature sequences of different scales. , the calculation formula is as follows:

[0068] ,

[0069] ,

[0070] ,

[0071] in, Indicates the The fission characteristic sequence of the scale, represents the fission characteristic sequence of the th scale, is the category label, is the fission characteristic sequence of different scales, is the number of fission characteristic sequences;

[0072] Fission signature sequence The input is fed into the VQ-VAE decoder to generate a fissile image that is similar to the target reference image.

[0073] (3) Generate initial features based on the vectorized representation of the target reference image, use the mask matrix to mark the area to be edited of the target reference image, and given the auxiliary target reference image of the home appliance product, extract the feature sequence of the auxiliary target reference image. Based on the initial features, the mask matrix marks and the feature sequence of the auxiliary target reference image, perform feature fusion scale by scale to obtain the editing feature sequence, decode and generate the editing image, and complete the personalized home appliance product design.

[0074] In the embodiment, in addition to providing a target reference image of a home appliance product, the user also provides multiple auxiliary target reference images of the home appliance product, such as Figure 5 As shown in the figure, the areas to be edited are marked in the target reference image. The areas selected by the user are represented by a mask matrix. The masking methods that can be used are binary masking or Gaussian masking. The binary mask includes: randomly selected editing areas marked as 1, indicating that the original content is retained; unselected editing areas are marked as 0, indicating that they are replaced with new content.

[0075] Then, image editing is performed based on the vectorized representation, realizing local precise replacement and providing users with customized modification of home appliance products. Specifically, initial editing features are generated based on the vectorized representation of the target reference image , the mask matrix is used to mark the to-be-edited region of the target reference image, a plurality of auxiliary target reference images of home appliance products are given, and the VQ-VAE encoder is used to decompose the auxiliary target reference images into a multi-scale feature sequence , represents the feature sequence of the i-th scale after decomposition of the auxiliary target reference image.

[0076] For the mask region in the initial feature, the current feature is retained.

[0077] For the non-mask region in the initial feature, the feature of the corresponding layer of the auxiliary target reference image is introduced.

[0078] The current feature and the feature of the corresponding layer of the auxiliary target reference image are fused to predict the next scale mask feature, and the mask feature is calculated by scale-by-scale weighting to obtain the editing feature sequence, and the calculation formula is as follows:

[0079] ,

[0080] ,

[0081] ,

[0082] wherein, represents the vectorized representation of the target reference image, is the category label, is the initial editing feature obtained by the vectorized representation of the target reference image, represents the editing feature sequence of the i-th scale, represents the editing feature sequence of different scales predicted based on the visual self-recurrence model, represents the number of editing feature sequences, and Mask is the mask matrix.

[0083] Finally, the output editing feature sequence is input into the VQ-VAE decoder to generate an edited image.

[0084] The effect of image editing is as shown in Figure 6 ​​As shown, when predicting the feature Token map of different scales backward, the VAR model is used to first perform a weighted sum with a mask on the feature map of the previous scale, so that the feature Token of the next scale can fuse the feature information of multiple reference maps, and then modify specific components according to user needs to achieve the purpose of personalized editing.

[0085] In another aspect, the embodiments also provide a personalized home appliance product design device based on a visual autoregressive model, comprising a memory and a processor, the memory is used to store a computer program, characterized in that the processor is used to realize the personalized home appliance product design method based on the visual autoregressive model when executing the computer program.

[0086] It should be noted that the personalized home appliance product design method and device based on the visual autoregressive model provided in the above embodiments should be illustrated by the division of the above functional modules when performing personalized design of home appliance products, and the above functions can be completed by different functional modules according to needs, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the personalized home appliance product design method based on the visual autoregressive model and the personalized home appliance product design device based on the visual autoregressive model provided in the above embodiments belong to the same concept, and the specific implementation process is described in the personalized home appliance product design method based on the visual autoregressive model. Embodiments, which will not be repeated here.

[0087] The above embodiments describe the technical solutions and advantages of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the present application. Any modification, supplement and equivalent replacement made within the principle range of the present application should be included in the protection scope of the present application.

Claims

1. A personalized home appliance product design method based on a visual autoregressive model, characterized in that: The following steps are involved: (1) Given a target reference image of a home appliance product, it is converted into a series of discrete token sequences through a visual autoregressive model, and a vectorized representation of the target reference image is generated based on iterative optimization of the discrete token sequence; (2) Add noise perturbation to the vectorized representation of the target reference image to generate a fission feature sequence, and decode it to generate a fission image similar to the target reference image; (3) Generate initial features based on the vectorized representation of the target reference image, use the mask matrix to mark the area to be edited of the target reference image, and given the auxiliary target reference image of the home appliance product, extract the feature sequence of the auxiliary target reference image. Based on the initial features, the mask matrix marks and the feature sequence of the auxiliary target reference image, perform feature fusion scale by scale to obtain the editing feature sequence, decode and generate the editing image, and complete the personalized home appliance product design.

2. The personalized home appliance product design method according to claim 1, characterized in that: Step (1) specifically includes: taking a target reference image of a home appliance product , which is encoded by the VQ-VAE encoder in the visual autoregressive model as Token sequences with different scales of discreteness As the true label, based on the discrete Token sequence, the vectorized representation of the target reference image is generated through iterative optimization using a training-free method .

3. The personalized home appliance product design method according to claim 2, characterized in that: When using the Training-free method, set the loss function to iteratively optimize and generate a vectorized representation of the target reference image. , the calculation formula is as follows: ， ， ， Among them, codebook represents the size of the Token codebook in the VQ-VAE encoder. Represents the discrete Token index variable in the codebook, represents the visual autoregressive model, Represents the first prediction based on the visual autoregressive model The characteristic sequence of the scale, is the category label, Represents feature sequences of different scales predicted based on the visual autoregressive model, is the number of feature maps.

4. The personalized home appliance product design method according to claim 3, characterized in that: Step (2) specifically includes: vectorized representation of the target reference image Inject Gaussian noise , based on the visual autoregressive model, the fission features of the next scale are predicted, and the fission features are fused scale by scale to obtain the fission feature sequences of different scales , the calculation formula is as follows: ， ， ， in, Indicates the The fission characteristic sequence of the scale, Indicates the The fission characteristic sequence of the scale, is the category label, are fission characteristic sequences of different scales, is the number of fission characteristic sequences; Fission signature sequence The input is fed into the VQ-VAE decoder to generate a fissile image that is similar to the target reference image.

5. The personalized home appliance product design method according to claim 1, characterized in that: In step (3), the use of a mask matrix to mark the area to be edited includes: randomly selecting the editing area on the target reference image, generating a binary mask or a Gaussian mask, and marking the area to be edited based on the binary mask or the Gaussian mask matrix.

6. The personalized home appliance product design method according to claim 5, characterized in that: The binary mask includes: the randomly selected editing area is marked as 1, which is used to indicate that the original content is retained, and the unselected editing area is marked as 0, which is used to indicate that it is replaced with new content.

7. The personalized home appliance product design method according to claim 6, characterized in that: In step (3), given an auxiliary target reference image of a home appliance product, extracting the feature sequence of the auxiliary target reference image includes: decomposing the auxiliary target reference image into a multi-scale feature sequence through a VQ-VAE encoder , represents the auxiliary target reference image after decomposition Feature sequence at layer scale.

8. The personalized home appliance product design method according to claim 7, characterized in that: In step (3), the feature sequence based on the initial features, the mask matrix mark and the auxiliary target reference image is subjected to feature fusion scale by scale to obtain the edited feature sequence, including: For the mask area in the initial feature, keep the current feature ; For the non-masked area in the initial feature, the features of the corresponding layer of the auxiliary target reference image are introduced ; The current feature and the features of the corresponding layer of the auxiliary target reference image After fusion, the mask features of the next scale are predicted, and the mask features are weighted and calculated scale by scale to obtain the edit feature sequence. The calculation formula is as follows: ， ， ， in, represents the vectorized representation of the target reference image, is the category label, is the initial editing feature obtained by vectorizing the target reference image, Indicates the The editing feature sequence of each scale, represents the visual autoregressive model, Represents the editing feature sequences of different scales predicted based on the visual autoregressive model, Represents the number of edit feature sequences, and Mask is the mask matrix.

9. The personalized home appliance product design method according to claim 8, characterized in that: The decoding step (3) generates the edited image, including: converting the output edit feature sequence Input to the VQ-VAE decoder to generate the edited image.

10. A personalized home appliance product design device based on a visual autoregressive model, comprising a memory and a processor, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the personalized home appliance product design method based on the visual autoregressive model described in any one of claims 1 to 9 when executing the computer program.

Images