Personalized household appliance product design method and device based on visual autoregression model

Abstract

The invention discloses a personalized household appliance product design method and device based on a visual autoregression model, and the method comprises the steps: giving a target reference image of a household appliance product, converting the target reference image into a series of discrete Token sequences through the visual autoregression model, and generating the vectorization representation of the target reference image based on the Token sequence iteration optimization; noise disturbance is added to the vectorized representation, a fission feature sequence is generated, and a fission image is generated through decoding; the method comprises the steps of generating initial editing features based on vectorized representation, marking a to-be-edited area of a target reference image by using a mask matrix, giving an auxiliary target reference image, extracting a feature sequence of the auxiliary target reference image, obtaining an editing feature sequence based on the initial features, the mask matrix mark and the feature sequence through scale-by-scale fusion, and decoding to generate an editing image. And personalized household appliance product design is completed. According to the invention, by constructing the visual autoregression model of the household appliance product, cooperative control of global disturbance and local editing is realized in the feature space, and the generation effect of the household appliance image is improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a method and device for designing personalized home appliance products based on a visual autoregressive model. Background Art

[0002] In the past two years, the field of image generation has garnered significant attention, particularly with the open-source release of the stable diffusion model and the continuous updates to the DALL-E series of models, which have collectively elevated the field to new heights. Despite the remarkable image generation capabilities of diffusion models, their application to tasks such as image editing faces significant challenges. This is primarily due to the potential for hallucinations in synthesized images, failure to strictly adhere to editing instructions, and inability to maintain consistency with the underlying image.

[0003] Given images of one or more concepts as input, personalized image generation aims to generate image variants of the given concept or identity. Methods such as DreamBooth and Textual Inversion extract relevant concepts from a set of images and regenerate new images by designing richer inversion spaces. Textual Inversion is a method that uses reverse learning to enable text-based graph models to better understand and generate specific concepts, freezing the parameters of the pre-trained model during training.

[0004] However, existing personalized image generation methods feed reference images, text or depth maps, line drawings, encoding information and other conditions into the generative model to generate the desired personalized image. Therefore, there are the following challenges: First, the effect of text-driven image editing is highly dependent on the user's description. If the instructions are ambiguous, the results generated by the model are often difficult to fully match the user's actual expectations. Secondly, for tasks that require fine operations, such as boundary processing and target replacement in complex occlusion scenes, existing methods have difficulty in achieving accurate and detailed editing. Finally, these methods usually rely on a large amount of high-quality training data and high-performance computing resources, and have high requirements for data quality and computing power, which brings certain barriers to practical application.

[0005] Autoregressive models are a type of generative model that treat text-to-image generation as a sequence-to-sequence modeling problem, similar to machine translation. Early pioneering work, such as VQ-VAE, introduced the concept of vector quantization, enabling images to be encoded as sequences of discrete tokens, similar to how language is processed. This approach allows models to process visual data in a manner similar to how they process language tokens. However, efforts to apply these models to conditional generation tasks have been relatively limited.

[0006] Other studies have attempted to improve autoregressive models by re-evaluating the next-token prediction paradigm, questioning the traditional line-by-line raster scanning approach to generate image tags, arguing that images require more global context than text. Inspired by masked autoencoders, MaskGIT uses an iterative mask modeling approach that learns to predict random masked tags by paying attention to other tags and iteratively decoding them during inference. The VAR visual autoregressive model shifts the next-token prediction paradigm to a coarse-to-fine next-scale prediction, significantly improving the visual quality of generated images.

[0007] In the field of home appliance design, current home appliance big models primarily include natural semantic big models and visual big models, or video big models. Unlike general-purpose big models, the training data for home appliance big models incorporates a wealth of mature marketing methodologies and marketing data. Furthermore, existing home appliance big models are largely based on self-constructed home appliance product datasets, fine-tuning the pre-trained big models. However, the quality and quantity of training datasets significantly impact the performance of these big models. Currently, there is a lack of training datasets for home appliance products.

[0008] Therefore, in summary, in order to solve the problems of missing training data sets in the field of home appliance design and the inability of traditional general large models to achieve more accurate personalized image generation, the present invention proposes a method for personalized design of home appliance products based on a visual autoregressive model. Summary of the Invention

[0009] The present invention provides a method and device for personalized design of home appliances based on a visual autoregressive model. Aiming at the field of personalized design of home appliances, the present invention provides a more accurate image personalized generation technology. When training data is relatively scarce, the technology uses a reference image of a home appliance to achieve coordinated control of global perturbation and local editing through feature space, thereby achieving detail replacement of different home appliances and improving the generation effect of home appliance images of large models.

[0010] A personalized home appliance product design method based on a visual autoregressive model includes the following steps: (1) Given a target reference image of a home appliance product, it is converted into a series of discrete token sequences through a visual autoregressive model, and a vectorized representation of the target reference image is generated based on iterative optimization of the discrete token sequence; (2) Add noise perturbation to the vectorized representation of the target reference image to generate a fission feature sequence, and decode it to generate a fission image similar to the target reference image; (3) Generate initial features based on the vectorized representation of the target reference image, use the mask matrix to mark the area to be edited of the target reference image, and given the auxiliary target reference image of the home appliance product, extract the feature sequence of the auxiliary target reference image. Based on the initial features, the mask matrix marks and the feature sequence of the auxiliary target reference image, perform feature fusion scale by scale to obtain the editing feature sequence, decode and generate the editing image, and complete the personalized home appliance product design.

[0011] In one embodiment, step (1) specifically includes: a target reference image of a home appliance product , which is encoded by the VQ-VAE encoder in the visual autoregressive model as Token sequences with different scales of discreteness As the true label, based on the discrete Token sequence, the vectorized representation of the target reference image is generated through iterative optimization using a training-free method .

[0012] In one embodiment, when using the training-free method, the loss function is set to iteratively optimize and generate a vectorized representation of the target reference image. , the calculation formula is as follows: , , , Among them, codebook represents the size of the Token codebook in the VQ-VAE encoder. Represents the discrete Token index variable in the codebook, represents the visual autoregressive model, Represents the first prediction based on the visual autoregressive model The characteristic sequence of the scale, is the category label, Represents feature sequences of different scales predicted based on the visual autoregressive model, is the number of feature maps.

[0013] In one embodiment, step (2) specifically includes: vectorizing the target reference image Inject Gaussian noise , based on the visual autoregressive model, the fission features of the next scale are predicted, and the fission features are fused scale by scale to obtain the fission feature sequences of different scales , the calculation formula is as follows: , , , in, Indicates the The fission characteristic sequence of the scale, Indicates the The fission characteristic sequence of each scale, is the category label, are fission characteristic sequences of different scales, is the number of fission characteristic sequences; Fission signature sequence The input is fed into the VQ-VAE decoder to generate a fissile image that is similar to the target reference image.

[0014] In one embodiment, in step (3), the use of a mask matrix to mark the area to be edited includes: randomly selecting the editing area on the target reference image, generating a binary mask or a Gaussian mask, and marking the area to be edited based on the binary mask or the Gaussian mask matrix.

[0015] In one embodiment, the binary mask includes: a randomly selected editing area is marked as 1, which is used to indicate that the original content is retained; an unselected editing area is marked as 0, which is used to indicate that it is replaced with new content.

[0016] In one embodiment, in step (3), given an auxiliary target reference image of a home appliance product, extracting a feature sequence of the auxiliary target reference image includes: decomposing the auxiliary target reference image into a multi-scale feature sequence through a VQ-VAE encoder , represents the auxiliary target reference image after decomposition Discrete Token feature map at layer scale.

[0017] In one embodiment, in step (3), the feature fusion based on the initial features, the mask matrix mark and the feature sequence of the auxiliary target reference image is scale-by-scale to obtain the edited feature sequence, including: For the mask area in the initial feature, keep the current feature ; For the non-masked area in the initial feature, the features of the corresponding layer of the auxiliary target reference image are introduced ; The current feature and the features of the corresponding layer of the auxiliary target reference image After fusion, the mask features of the next scale are predicted, and the mask features are weighted and calculated scale by scale to obtain the edit feature sequence. , the calculation formula is as follows: , , , in, represents the vectorized representation of the target reference image, is the category label, is the initial editing feature obtained by vectorizing the target reference image, Indicates the The editing feature sequence of each scale, Represents the editing feature sequences of different scales predicted based on the visual autoregressive model, Represents the number of edit feature sequences, and Mask is the mask matrix.

[0018] In one embodiment, the decoding step (3) to generate the edited image includes: Input to the VQ-VAE decoder to generate the edited image.

[0019] On the other hand, the present invention also provides a personalized home appliance product design device based on a visual autoregressive model, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is characterized in that when executing the computer program, the processor is used to implement the personalized home appliance product design method based on the visual autoregressive model.

[0020] Compared with the prior art, the present invention has the following beneficial effects: (1) By combining textual inversion with a visual autoregressive model, a visual autoregressive model architecture dedicated to home appliances is constructed. The vectorized representation is optimized in a training-free manner to adapt to data-scarce scenarios in the home appliance field.

[0021] (2) The reference image of the home appliance product is converted into embedded features. Image fission and image editing are tightly coupled through feature space operations to achieve local fusion of multiple home appliance product images and replacement of details of different home appliance products, avoiding the instability of pixel-level operations and achieving precise and controllable semantic-level transformation and control.

[0022] (3) Relying on pre-trained models, an efficient vectorized representation of home appliance products is obtained through cross-entropy optimization, which can realize the style transfer of home appliance products with only a small number of example samples. Compared with traditional fine-tuning methods, it reduces the training data requirements by 98%, saving training resources. Especially in application scenarios with high requirements for details such as home appliances, it can accurately replace and edit local details of the product, ensuring that the final effect of the home appliance product is consistent with the design expectations, greatly expanding the application scope of high-quality image editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1This is an overview of the personalized home appliance product design process based on a visual autoregressive model provided by an embodiment of the present invention.

[0024] Figure 2 A flowchart of a personalized home appliance product design method based on a visual autoregressive model provided by an embodiment of the present invention.

[0025] Figure 3 A flowchart of iteratively optimizing the vectorized representation of a target reference image provided by an embodiment of the present invention.

[0026] Figure 4 The effect diagram of personalized home appliance product image reconstruction and fission based on the visual autoregressive model provided by the present invention.

[0027] Figure 5 This is a flowchart of mask fusion based on multiple images provided by an embodiment of the present invention.

[0028] Figure 6 This is an effect diagram of personalized home appliance product image editing based on a visual autoregressive model provided by the present invention. DETAILED DESCRIPTION

[0029] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0030] In order to solve the problems of missing training data sets and the inability of traditional general large models to achieve more accurate personalized image generation in the field of home appliance design, this paper proposes a personalized home appliance product design method based on a visual autoregressive model. Figure 1 As shown in the figure, the home appliance product image is vectorized based on the visual autoregressive model, that is, the entire image is restored with a vector; image fission and image editing are respectively implemented based on the image vectorization to complete the personalized home appliance product design. Figure 2 The specific implementation steps are as follows: (1) Given a target reference image of a home appliance product, it is converted into a series of discrete Token sequences through a visual autoregressive model, and a vectorized representation of the target reference image is generated based on iterative optimization of the discrete Token sequence.

[0031] In the embodiment, a visual autoregressive model is selected as the basic model. The autoregressive model is a type of probabilistic model that decomposes complex data distribution into the product of conditional probabilities, thereby generating target data in sequence. In the visual autoregressive model, the visual image data must first be converted into a series of discrete tokens. After representing the visual image content as an ordered sequence of tokens, the autoregressive model predicts the token sequence of the next scale through the token sequences of all previous scales. This process can be expressed by the following formula: , in, It means predicting the probability distribution of the current token given a historical token sequence; A discrete Token sequence representing the entire visual image, Indicates the discrete Token, i represents the index variable of the discrete Token sequence, Indicates the number of discrete Token sequence index variables, Indicates the The known Token sequence before the position, represents the parameters of the autoregressive model.

[0032] The visual autoregressive model consists of the following modules: a VQ-VAE encoder, a VQ-VAE decoder, and an autoregressive model. The VQ-VAE encoder encodes visual image data into discrete token sequences of different scales; the VQ-VAE decoder restores token sequences of different scales back to visual data; and the autoregressive model predicts the token sequence of the next scale.

[0033] Next, the user provides a reference image, and the visual autoregressive model continuously iteratively optimizes a vector, which can be reversely reconstructed to restore the entire reference image. Specifically: like Figure 3 As shown, a target reference image of a home appliance product , which is encoded by the VQ-VAE encoder in the visual autoregressive model as Token sequences with different scales of discreteness As the true label, based on the discrete Token sequence, the vectorized representation of the target reference image is generated through iterative optimization using a training-free method The purpose of this vectorized representation is to feed it into the visual autoregressive model to restore the target reference image.

[0034] The Textual Inversion method is applied to the visual autoregressive model to build a personalized home appliance generation model. By freezing the pre-trained model parameters, the vectorized representation is optimized in a training-free manner. , adapted to the data scarcity scenario in the home appliance field, the loss function is expressed by cross entropy, and the calculation formula is as follows: , , , Among them, codebook represents the size of the Token codebook in the VQ-VAE encoder. Represents the discrete Token index variable in the codebook, represents the visual autoregressive model, Represents the first prediction based on the visual autoregressive model The characteristic sequence of the scale, is the category label, Represents feature sequences of different scales predicted based on the visual autoregressive model, is the number of feature maps; By calculating the characteristic sequence and the true label Cross entropy between, iterative optimization .

[0035] (2) Noise perturbation is added to the vectorized representation of the target reference image to generate a fission feature sequence, which is then decoded to generate a fission image similar to the target reference image.

[0036] In this embodiment, image fission is to generate a similar image from a target reference image, and this function can be achieved based on the vectorized representation of the image. Specifically: Vectorized representation of the target reference image Inject Gaussian noise , so that from The predicted multi-scale feature sequence is no longer a simple restoration of the original image, but can obtain similar variants of the target reference image, thus achieving global fine-tuning of home appliance products, such as Figure 4 As shown, it can generate images that are similar to the reference image but also have certain differences.

[0037] Based on the visual autoregressive model VAR, the fission features of the next scale are predicted, and the fission features are fused scale by scale to obtain the fission feature sequences of different scales. , the calculation formula is as follows: , , , in, Indicates the The fission characteristic sequence of each scale, represents the fission characteristic sequence of the th scale, is the category label, are fission characteristic sequences of different scales, is the number of fission characteristic sequences; Fission signature sequence The input is fed into the VQ-VAE decoder to generate a fissile image that is similar to the target reference image.

[0038] (3) Generate initial features based on the vectorized representation of the target reference image, use the mask matrix to mark the area to be edited of the target reference image, and given the auxiliary target reference image of the home appliance product, extract the feature sequence of the auxiliary target reference image. Based on the initial features, the mask matrix marks and the feature sequence of the auxiliary target reference image, perform feature fusion scale by scale to obtain the editing feature sequence, decode and generate the editing image, and complete the personalized home appliance product design.

[0039] In the embodiment, in addition to providing a target reference image of a home appliance product, the user also provides multiple auxiliary target reference images of the home appliance product, such as Figure 5 As shown, the target reference image is marked with areas to be edited. The areas selected by the user are represented by a mask matrix. The masking methods used include binary masking and Gaussian masking. The binary masking includes: randomly selected edit areas marked as 1, indicating that the original content is retained; unselected edit areas are marked as 0, indicating that they are replaced with new content.

[0040] Then, image editing is performed based on the vectorized representation to achieve accurate local replacement and provide users with customized transformation of home appliances. Specifically: initial editing features are generated based on the vectorized representation of the target reference image , use the mask matrix to mark the target reference image to be edited area, given multiple auxiliary target reference images of home appliances, through the VQ-VAE encoder decomposition into multi-scale feature sequences , represents the auxiliary target reference image after decomposition Feature sequence at layer scale.

[0041] For the mask area in the initial feature, keep the current feature ; For the non-masked area in the initial feature, the features of the corresponding layer of the auxiliary target reference image are introduced ; The current feature and the features of the corresponding layer of the auxiliary target reference image After fusion, the mask features of the next scale are predicted, and the mask features are weighted and calculated scale by scale to obtain the edit feature sequence. The calculation formula is as follows: , , , in, represents the vectorized representation of the target reference image, is the category label, is the initial editing feature obtained by vectorizing the target reference image, Indicates the The editing feature sequence of each scale, Represents the editing feature sequences of different scales predicted based on the visual autoregressive model, Represents the number of edit feature sequences, and Mask is the mask matrix.

[0042] Finally, the output edit feature sequence Input to the VQ-VAE decoder to generate the edited image.

[0043] Image editing effects such as Figure 6 As shown in the figure, when using the VAR model to predict the feature token images of different scales, a masked weighted sum is first performed on the feature map of the previous scale. In this way, the feature token of the next scale will integrate the feature information of multiple reference images, and then specific components can be modified according to user needs to achieve the purpose of personalized editing.

[0044] On the other hand, an embodiment also provides a personalized home appliance product design device based on a visual autoregressive model, comprising a memory and a processor, wherein the memory is used to store a computer program, and wherein the processor is used to implement the personalized home appliance product design method based on a visual autoregressive model when executing the computer program.

[0045] It should be noted that the personalized home appliance product design method and device based on the visual autoregressive model provided in the above embodiment should be illustrated by the division of the above-mentioned functional modules when performing personalized design of home appliances. The above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the personalized home appliance product design method based on the visual autoregressive model provided in the above embodiment and the personalized home appliance product design device based on the visual autoregressive model are of the same concept. The specific implementation process is detailed in the embodiment of the personalized home appliance product design method based on the visual autoregressive model, which will not be repeated here.

[0046] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A personalized home appliance product design method based on a visual autoregressive model, characterized in that: The following steps are involved: (1) Given a target reference image of a home appliance product, it is converted into a series of discrete token sequences through a visual autoregressive model, and a vectorized representation of the target reference image is generated based on iterative optimization of the discrete token sequence; (2) Add noise perturbation to the vectorized representation of the target reference image to generate a fission feature sequence, and decode it to generate a fission image similar to the target reference image; (3) Generate initial features based on the vectorized representation of the target reference image, use the mask matrix to mark the area to be edited of the target reference image, and given the auxiliary target reference image of the home appliance product, extract the feature sequence of the auxiliary target reference image. Based on the initial features, the mask matrix marks and the feature sequence of the auxiliary target reference image, perform feature fusion scale by scale to obtain the editing feature sequence, decode and generate the editing image, and complete the personalized home appliance product design.

2. The personalized home appliance product design method according to claim 1, characterized in that: Step (1) specifically includes: taking a target reference image of a home appliance product , which is encoded by the VQ-VAE encoder in the visual autoregressive model as Token sequences with different scales of discreteness As the true label, based on the discrete Token sequence, the vectorized representation of the target reference image is generated through iterative optimization using a training-free method .

3. The personalized home appliance product design method according to claim 2, characterized in that: When using the Training-free method, set the loss function to iteratively optimize and generate a vectorized representation of the target reference image. , the calculation formula is as follows: ， ， ， Among them, codebook represents the size of the Token codebook in the VQ-VAE encoder. Represents the discrete Token index variable in the codebook, represents the visual autoregressive model, Represents the first prediction based on the visual autoregressive model The characteristic sequence of the scale, is the category label, Represents feature sequences of different scales predicted based on the visual autoregressive model, is the number of feature maps.

4. The personalized home appliance product design method according to claim 3, characterized in that: Step (2) specifically includes: vectorized representation of the target reference image Inject Gaussian noise , based on the visual autoregressive model, the fission features of the next scale are predicted, and the fission features are fused scale by scale to obtain the fission feature sequences of different scales , the calculation formula is as follows: ， ， ， in, Indicates the The fission characteristic sequence of the scale, Indicates the The fission characteristic sequence of the scale, is the category label, is the fission characteristic sequence of different scales, is the number of fission characteristic sequences; Fission signature sequence The input is fed into the VQ-VAE decoder to generate a fissile image that is similar to the target reference image.

5. The personalized home appliance product design method according to claim 1, characterized in that: In step (3), the use of a mask matrix to mark the area to be edited includes: randomly selecting the editing area on the target reference image, generating a binary mask or a Gaussian mask, and marking the area to be edited based on the binary mask or the Gaussian mask matrix.

6. The personalized home appliance product design method according to claim 5, characterized in that: The binary mask includes: the randomly selected editing area is marked as 1, which is used to indicate that the original content is retained, and the unselected editing area is marked as 0, which is used to indicate that it is replaced with new content.

7. The personalized home appliance product design method according to claim 6, characterized in that: In step (3), given an auxiliary target reference image of a home appliance product, extracting the feature sequence of the auxiliary target reference image includes: decomposing the auxiliary target reference image into a multi-scale feature sequence through a VQ-VAE encoder , represents the auxiliary target reference image after decomposition Feature sequence at layer scale.

8. The personalized home appliance product design method according to claim 7, characterized in that: In step (3), the feature sequence based on the initial features, the mask matrix mark and the auxiliary target reference image is subjected to feature fusion scale by scale to obtain the edited feature sequence, including: For the mask area in the initial feature, keep the current feature ; For the non-masked area in the initial feature, the features of the corresponding layer of the auxiliary target reference image are introduced ; The current feature and the features of the corresponding layer of the auxiliary target reference image After fusion, the mask features of the next scale are predicted, and the mask features are weighted and calculated scale by scale to obtain the edit feature sequence. The calculation formula is as follows: ， ， ， in, represents the vectorized representation of the target reference image, is the category label, is the initial editing feature obtained by vectorizing the target reference image, Indicates the The editing feature sequence of each scale, Represents the editing feature sequences of different scales predicted based on the visual autoregressive model, Represents the number of edit feature sequences, and Mask is the mask matrix.

9. The personalized home appliance product design method according to claim 8, characterized in that: The decoding step (3) generates the edited image, including: converting the output edit feature sequence Input to the VQ-VAE decoder to generate the edited image.

10. A personalized home appliance product design device based on a visual autoregressive model, comprising a memory and a processor, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the personalized home appliance product design method based on the visual autoregressive model described in any one of claims 1 to 9 when executing the computer program.

Images