Methods, devices, computer equipment and storage media for generating pet visual data

Abstract

This application provides a method, apparatus, computer device, and storage medium for generating pet visual data. The method includes receiving a data generation request, which includes visual cue text and reference visual data of a target pet; invoking a pre-trained generative model based on the data generation request, wherein the generative model includes a data fusion sub-model and a visual processing sub-model; invoking the data fusion sub-model to perform data fusion based on the visual cue text and the reference visual data to obtain a visual latent vector; and invoking the visual processing sub-model to generate visual data based on the visual latent vector to obtain target visual data corresponding to the visual cue text, wherein the target visual data includes an image and / or a video of the target pet. This method can improve the consistency of pet visual data generation.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to a method, apparatus, computer device, and storage medium for generating pet visual data. Background Technology

[0002] With the development of IoT and AI technologies, users can leverage AI to generate personalized content featuring their pets. Whether it's still images or videos, these can capture the unique characteristics of their pets, fulfilling users' needs for recording and sharing.

[0003] Currently, pet image generation and pet video generation typically employ different models or systems, which operate independently. For example, an image generation model generates personalized pet images, while a video generation model generates personalized pet videos. However, this approach often results in poor consistency between the generated images and videos in terms of style and other aspects. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, computer device, and storage medium for generating pet visual data, aiming to improve the consistency of pet visual data generation.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for generating pet visual data, comprising:

[0006] Receive a data generation request, which includes visual cue text and reference visual data of the target pet;

[0007] The pre-trained generative model is invoked based on the data generation request. The generative model includes a data fusion sub-model and a visual processing sub-model.

[0008] The data fusion sub-model is invoked to perform data fusion based on visual cue text and reference visual data to obtain a visual latent vector;

[0009] The visual processing sub-model is invoked to generate visual data based on the visual latent vector, resulting in target visual data corresponding to the visual cue text. The target visual data includes target pet images and / or target pet videos.

[0010] The training process of the generative model includes:

[0011] Obtain multimodal training samples of the sample pets, including sample pet images and sample pet videos;

[0012] Image pre-training of the visual processing sub-model based on sample pet images;

[0013] After image pre-training is completed, the visual processing sub-model is jointly trained based on sample pet images and sample pet videos.

[0014] In some embodiments, the sample pet image includes a first pet image and a second pet image, wherein the resolution of the first pet image is smaller than that of the second pet image;

[0015] Image pre-training of the visual processing sub-model based on sample pet images includes:

[0016] Noise processing is applied to the first and second pet images;

[0017] The visual processing sub-model is initially trained using the first pet image after noise processing.

[0018] After the initial image training is completed, a hybrid image set is constructed based on the first pet image after noise processing and the second pet image after noise processing;

[0019] The visual processing sub-model is trained at mixed scales based on a mixed image set.

[0020] In some embodiments, initial image training of the visual processing sub-model is performed based on a noise-processed first pet image, including:

[0021] The visual processing sub-model is invoked to perform denoising processing on the first pet image after noise processing to obtain the predicted noise speed;

[0022] The parameters of the visual processing sub-model are adjusted based on the predicted noise velocity and the actual noise velocity of the first pet image after noise processing.

[0023] In some embodiments, the sample pet videos include a first pet video, a second pet video, and a third pet video. The first pet video is a pet video with a resolution within a preset first resolution range and a video duration within a preset first duration range. The second pet video is a pet video with a resolution within the first resolution range and a video duration within a preset second duration range, where the lower limit of the second duration range is greater than the upper limit of the first duration range. The third pet video is a pet video with a resolution within the preset second resolution range and a video duration within the second duration range, where the lower limit of the second resolution range is greater than the upper limit of the first resolution range. The visual joint training includes a first-stage training, a second-stage training, and a third-stage training.

[0024] The visual processing sub-model is jointly trained based on sample pet images and sample pet videos, including:

[0025] The visual processing sub-model is trained in the first stage based on sample pet images and the first pet video.

[0026] The visual processing sub-model is trained in the second stage based on sample pet images and second pet videos.

[0027] The visual processing sub-model is trained in the third stage based on sample pet images and third-party pet videos.

[0028] In some embodiments, the data fusion sub-model includes a text encoder and an autoregressive encoder, and the visual processing sub-model includes an image encoder and a visual processing network.

[0029] The method also includes:

[0030] The image encoder is invoked to extract visual features from the reference visual data to obtain the target image vector;

[0031] The data fusion sub-model is invoked to perform data fusion based on visual cue text and reference visual data to obtain visual latent vectors, including:

[0032] The text encoder is invoked to extract semantic features from the visual cues, resulting in the target text vector;

[0033] An autoregressive encoder is invoked to perform attention fusion on the target text vector and the target image vector to obtain the visual latent vector;

[0034] The visual processing sub-model is invoked to generate visual data based on the visual latent vector, resulting in target visual data corresponding to the visual cue text, including:

[0035] The visual processing network is invoked to generate visual data from the visual latent vectors, thus obtaining the target visual data.

[0036] In some embodiments, the visual processing network includes an autoregressive visual decoder and a stream matching decoder;

[0037] The visual processing network is invoked to generate visual data from the latent visual vectors, resulting in target visual data, including:

[0038] The autoregressive visual decoder is invoked to decode the visual latent vectors, obtaining preliminary visual data;

[0039] The stream matching decoder is invoked to denoise the preliminary visual data to obtain the target visual data, where the resolution of the target visual data is greater than that of the preliminary visual data.

[0040] In some embodiments, the training process of the generative model further includes:

[0041] Get sample prompt text;

[0042] Construct training sample pairs based on sample pet images and sample prompt text, or sample pet videos and sample prompt text;

[0043] After the visual joint training is completed, the data fusion sub-model and the visual processing sub-model are jointly trained as a whole based on the training samples.

[0044] In some embodiments, receiving a data generation request includes:

[0045] Receive visual cues and the target pet's pet identification information;

[0046] Historical visual data is extracted from the corresponding pet media database based on the pet's identity information;

[0047] In response to the selection operation of historical visual data, a reference visual data is determined.

[0048] To achieve the above objectives, a second aspect of this application provides a pet visual data generation apparatus, the apparatus comprising:

[0049] The request acquisition unit is used to receive data generation requests, which include visual cue text and reference visual data of the target pet.

[0050] The model invocation unit is used to invoke a pre-trained generative model based on a data generation request. The generative model includes a data fusion sub-model and a visual processing sub-model.

[0051] The data fusion unit is used to call the data fusion sub-model to perform data fusion based on visual cue text and reference visual data to obtain visual latent vectors;

[0052] The visual processing unit is used to call the visual processing sub-model to generate visual data based on the visual latent vector, and obtain the target visual data corresponding to the visual cue text. The target visual data includes the target pet image and / or the target pet video.

[0053] The model training unit, used to train the generative model, includes:

[0054] The sample acquisition subunit is used to acquire multimodal training samples of sample pets, including sample pet images and sample pet videos;

[0055] The image pre-training subunit is used to pre-train the visual processing sub-model based on sample pet images.

[0056] The visual joint training subunit is used to perform visual joint training on the visual processing submodel based on sample pet images and sample pet videos after image pre-training.

[0057] To achieve the above objectives, a third aspect of this application provides a computer device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the pet visual data generation method described in the first aspect.

[0058] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the pet visual data generation method described in the first aspect.

[0059] To achieve the above objectives, a fifth aspect of this application provides a computer program product comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the pet visual data generation method described in the first aspect.

[0060] The pet visual data generation method provided in this application employs a unified model framework for image pre-training and joint visual training, enabling the generation model to master both static image generation and dynamic video generation capabilities. Therefore, in practical applications, the generation model can simultaneously support the generation of both static images and dynamic videos, improving the consistency between image and video generation. Attached Figure Description

[0061] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0062] Figure 1 A schematic diagram of the system used in the pet visual data generation method provided in this application;

[0063] Figure 2 A flowchart illustrating the pet visual data generation method provided in this application;

[0064] Figure 3 A schematic diagram of the structure of the generative model provided in this application;

[0065] Figure 4 Another flowchart illustrating the pet visual data generation method provided in this application;

[0066] Figure 5 A schematic diagram of the pet visual data generation device provided in this application;

[0067] Figure 6 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0069] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows:

[0070] Autoregressive networks (ARNs) are a type of neural network that models sequential data. The core idea of ​​an ARN is to learn the dependencies between elements in a data sequence and use historical information to predict the next element. Specifically, for a sequence (such as characters in text or values ​​in a time series), an ARN assumes that the probability of generating the current element depends only on all elements that precede it. By iterating this prediction process, it can generate a complete sequence. For example, in language models, an ARN predicts the next possible word based on preceding words, then adds the generated word to historical information to continue prediction, thus generating a coherent sentence.

[0071] Flow matching is a generative modeling technique designed to directly learn the transformation paths between data distributions. Unlike traditional generative models (such as GANs or VAEs), flow matching does not rely on explicit probability density estimation or adversarial training. Instead, it constructs a continuous vector field that gradually "flows" a simple distribution (such as a Gaussian distribution) to the target data distribution. Specifically, flow matching defines a time-varying vector field that allows samples starting from a noisy distribution to evolve along this vector field and eventually reach the true data distribution. During training, the model learns to optimize this vector field so that the sample transformation between any two time points matches the observed data changes.

[0072] With the development of IoT and AI technologies, users can leverage AI to generate personalized content featuring their pets. Whether it's still images or videos, these can capture the unique characteristics of pets, fulfilling users' needs for recording and sharing. However, the methods for generating pet visual data (i.e., images and videos) in these technologies suffer from the following problems:

[0073] 1. Disjointed Image and Video Generation. In related technologies, pet image generation and pet video generation typically employ different models or systems. These models or systems are independent of each other, resulting in a disconnect between image generation methods and video generation methods. This fragmented generation approach not only increases the complexity of system development and user experience but also makes it difficult to guarantee the uniformity and consistency of the generated pet images and videos in terms of style, appearance, etc.

[0074] Specifically, the main reasons why pet image generation and pet video generation use different models or systems are as follows:

[0075] (1) Temporal modeling threshold: Images only need to process a single frame, while videos also need to achieve cross-frame consistency and motion coherence (i.e., ensure that identity / texture does not drift or flicker). Based on this, it is often unstable to directly extend the single-frame model to video.

[0076] (2) Incompatibility between architecture and sampling: Images are mostly 2D diffusion / autoregressive single-frame sampling, while videos require temporal attention / 3D convolution and cross-frame constrained sampling. Based on this, hard unification will lead to degradation or excessive latency / memory.

[0077] (3) Data and computing power constraints: High-quality video data is scarce and expensive, which leads to a multiplied increase in computing / memory requirements for training and inference per frame. In addition, unified training of images and videos is prone to being dominated by image data, resulting in degradation or instability of video capabilities.

[0078] (4) Pet image generation and pet video generation are both complex tasks. The industry has long adopted the approach of “two models and two links” to conduct research, which is a legacy issue.

[0079] 2. Lack of personalized generation capabilities. While text-to-image (or video) models in related technologies can generate pet images (or videos) based on general descriptions, they struggle to precisely customize them for a specific pet. Secondly, even when users provide historical photos or videos of the pet as references, existing models still cannot generate new images or videos that are highly similar in appearance to that pet. Furthermore, existing models lack sufficient processing for the fusion of multimodal cues (such as image, video, and text cues), resulting in the generation results failing to capture the pet's detailed features and reducing the similarity of the generated content.

[0080] 3. Limited output size and style. Existing generative models typically have fixed output resolutions (e.g., 512×512 pixels) and duration limitations, making it difficult to meet users' personalized needs in terms of size, aspect ratio, and video length. Furthermore, existing technologies are also lacking in stylistic diversity and the ability to flexibly change background scenes, further restricting users' creative freedom.

[0081] Based on this, to address the issues of fragmented image and video generation in related technologies, this application provides a method, apparatus, computer device, and storage medium for generating pet visual data. This method employs a unified generation model architecture, capable of simultaneously supporting the generation of static images and dynamic videos. The pet visual data generation method provided in this application embodiment will be described below.

[0082] Reference Figure 1 This is a schematic diagram of the system used in the pet visual data generation method provided in this application. Figure 1 As shown, the system includes at least one smart device 110, at least one pet management client 130 corresponding to the smart device 110, and a pet device management server 120. The smart device 110 can be a smart litter box, smart feeder, monitoring camera, or smart water dispenser, etc. The pet management client 130 is a terminal that runs a pet management application. This terminal can be a mobile terminal, such as a smartphone, or a tablet, personal computer, headset, or vehicle terminal, etc. The pet management application can control smart devices such as smart litter boxes, smart feeders, and smart water dispensers. The pet device management server 120 is a server that can provide management, data processing, and interaction services to multiple pet management clients 130. The pet device management server 120 can be a single high-performance computer, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines) within a network platform. The pet device management server 120 can be a physical server or a cloud server.

[0083] Reference Figure 2 This is a flowchart illustrating the pet visual data generation method provided in this application.

[0084] In some embodiments, the pet visual data generation method provided in this application includes, but is not limited to, steps S201 to S204.

[0085] Step S201: Receive a data generation request, which includes visual cue text and reference visual data of the target pet;

[0086] Step S202: Invoke the pre-trained generative model according to the data generation request;

[0087] Step S203: The data fusion sub-model is invoked to perform data fusion based on the visual cue text and reference visual data to obtain the visual latent vector;

[0088] Step S204: The visual processing sub-model is invoked to generate visual data based on the visual latent vector, thereby obtaining target visual data corresponding to the visual cue text. The target visual data includes the target pet image and / or the target pet video.

[0089] In step S201 of some embodiments, the data generation request may refer to a request to generate visual data. The data generation request may include visual cue text and reference visual data of the target pet. The visual cue text may refer to text describing the content (such as scenes, actions, effects, etc.) corresponding to the expected final generated visual data (i.e., images or videos). The reference visual data may refer to reference images and / or reference videos of the target pet. It is understood that the target pet may refer to a pet that has been pre-bound or registered. For example, in a home setting, the target user can pre-register the target pet in a pet management application. After successful registration, images, videos, and other data of the target pet can be obtained from the smart device and stored in a pet media database. That is, smart devices such as smart litter boxes are equipped with camera modules, which can capture images or videos of the target pet's daily behavior. The reference visual data may refer to real pet images and / or videos determined from the pet media database that represent the desired creation or modification.

[0090] Specifically, target users can send data generation requests through the AI ​​assistant interface of the pet management application on the terminal.

[0091] In some embodiments, step S201 may include, but is not limited to, the following steps:

[0092] Receive visual cues and the target pet's pet identification information;

[0093] Historical visual data is extracted from the corresponding pet media database based on the pet's identity information;

[0094] In response to a selection operation based on historical visual data, reference visual data is determined.

[0095] In some embodiments, pet identity information may refer to the identity information of the target pet, such as the nickname entered in the pet management application when registering the target pet. It is understood that the target user can register multiple pets in the pet management application. Thus, historical visual data corresponding to the target pet can be extracted from the pet media database based on the pet identity information. Historical visual data may refer to historical images and / or videos of the target pet collected by a smart device. The target user can select representative (or any one or more) images and / or videos from the historical visual data as reference visual data for the target pet. For example, after determining the historical visual data, the terminal can display this historical visual data. Thus, reference visual data can be determined in response to the target user's selection of certain visual data from this historical visual data.

[0096] Understandably, target users can provide feedback and corrections to the images and / or videos captured by smart devices, thereby improving the accuracy of the reference visual data.

[0097] This application's embodiment uses a method for determining reference visual data based on a pet media database, eliminating the need for users to upload additional images or videos, thus simplifying user operations. Furthermore, since the pet media database stores daily images and videos of pets collected by smart devices, the reference visual data determined from this database is more representative, thereby improving the realism and accuracy of subsequently generated images and / or videos.

[0098] In step S202 of some embodiments, the generative model can be an artificial intelligence model that learns features from a large number of real images and / or videos in advance, thereby creating entirely new pet images and / or videos. Visual cue text and reference visual data are used as input data to the generative model, thus obtaining new images / videos that are similar to the content described by the visual cue text. For example, the reference visual data could be an image of the target pet lying on a windowsill, and the visual cue text could be "Generate a video of my pet cat playing in the garden," thus generating a video of the target pet playing in the garden based on the generative model.

[0099] In steps S203 to S204 of some embodiments, the generation model may include a data fusion sub-model and a visual processing sub-model. The data fusion sub-model may perform data fusion based on visual cue text and reference visual data to obtain a latent representation (i.e., a visual latent vector) of the image and / or video to be generated. The visual processing sub-model may further process this latent representation, such as performing encoding and decoding operations, to obtain the new image and / or video to be generated (i.e., target visual data).

[0100] Reference Figure 3 In some embodiments, the data fusion submodule includes a text encoder and an autoregressive encoder, and the visual processing submodel includes an image encoder and a visual processing network 301. The pet visual data generation method provided in this application includes, but is not limited to, the following steps:

[0101] Step S201: Receive a data generation request, which includes visual cue text and reference visual data of the target pet;

[0102] Step S202: Invoke the pre-trained generative model based on the visual cue text and reference visual data;

[0103] Step S301: Call the image encoder to extract visual features from the reference visual data to obtain the target image vector;

[0104] Step S2031: Call the text encoder to extract semantic features from the visual prompt text to obtain the target text vector;

[0105] Step S2032: Call the autoregressive encoder to perform attention fusion on the target text vector and the target image vector to obtain the visual latent vector;

[0106] Step S2041: Call the visual processing network to generate visual data from the visual latent vectors to obtain the target visual data.

[0107] Steps S201 and S202 have been described above and will not be repeated here.

[0108] In step S301 of some embodiments, reference visual data of the target pet is used as input data of the image encoder to extract visual features based on the reference visual data, thereby obtaining an image feature vector (i.e., the target image vector).

[0109] Furthermore, in some embodiments, such as Figure 3 As shown, at least one target cue image (or video) can also be acquired, and this target cue image and reference visual data can be used as input data for an image encoder to obtain a target image vector. The target cue image serves a similar purpose to the visual cue text; that is, the target cue image refers to the image used to describe the content (such as scene, action, effect, etc.) corresponding to the expected final generated visual data (i.e., image or video). For example, when the visual cue text indicates "Generate a video of my pet cat playing in the garden," the target cue image could be an image of a "garden." This limits the style of the garden in the final generated visual data.

[0110] In step S2031 of some embodiments, the visual cue text is used as input data for the text encoder to convert the visual cue text into a high-dimensional semantic vector representation based on the text encoder, thereby obtaining the target text vector.

[0111] In step S2032 of some embodiments, the output of the image encoder (i.e., the target image vector) and the output of the text encoder (i.e., the target text vector) are used as input data for the autoregressive encoder. The autoregressive encoder is a multimodal autoregressive Transformer encoder. The Transformer fuses the target image vector and the target text vector through a self-attention mechanism and generates a latent representation (i.e., a low-resolution visual latent vector) for the next output block (image or video frame sequence). Specifically, the self-attention mechanism dynamically calculates the correlation weights between the target image vector and the target text vector, enabling the model to adaptively capture meaningful semantic correspondences between the two modalities, thereby achieving more accurate cross-modal feature fusion. The fused multimodal representation is further used to generate the low-resolution visual latent vector corresponding to the next output block.

[0112] In step S2041 of some embodiments, the visual latent vector is used as input data of the visual processing network 301. The visual processing network 301 can decode based on the visual latent vector to obtain the corresponding image or video frame sequence, and then obtain the final image or video (i.e. target visual data) to be generated.

[0113] In some embodiments, such as Figure 3 As shown, the visual processing network 301 includes an autoregressive visual decoder and a stream matching decoder. Step S2041 includes, but is not limited to, the following sub-steps:

[0114] The autoregressive visual decoder is invoked to decode the visual latent vectors, obtaining preliminary visual data;

[0115] The stream matching decoder is invoked to denoise the initial visual data, thus obtaining the target visual data.

[0116] In this embodiment, the visual latent vector output by the autoregressive encoder is used as the input data of the autoregressive visual decoder. The autoregressive visual decoder decodes the visual latent vector into coarse, low-resolution visual data (i.e., preliminary visual data). Then, this low-resolution visual data is input to the decoder for progressive denoising and refinement to obtain high-resolution visual data, thereby generating a high-resolution final pet image or video (i.e., target visual data).

[0117] As described above, the generation model provided in this application incorporates autoregressive and stream matching techniques. Autoregressive generation features chunked generation. Chunked generation refers to the generation model dividing the content to be generated into several chunks and generating each chunk sequentially, with the generation of each chunk based on previously generated content. Thus, the generation module can flexibly expand the output size or duration by increasing the number of chunks, without being limited by a predetermined length. Therefore, in this application embodiment, it supports target users in submitting generation requests with arbitrary size, aspect ratio, and video duration, solving the problem of limited output size in related technologies.

[0118] Secondly, multi-scale processing is achieved by combining stream matching technology, meaning that it can generate content step-by-step according to different resolution levels. Specifically, the embodiments of this application can first generate low-resolution basic content (for capturing global patterns and structures), and then gradually add details at higher resolution layers. This multi-scale generation mechanism enables the generation model to generate images and / or videos from coarse to fine, while ensuring consistent content continuity across different scales. In other words, the generation model can generate image or video content block by block, continuously adding existing blocks to expand the screen size or video duration. Simultaneously, a multi-scale generation strategy from low to high resolution is adopted, first generating a basic low-resolution structure, and then gradually adding details to the target resolution. This design allows the generation model to be unrestricted by a fixed output size, flexibly generating visual content of any size and duration as needed. Thus, regardless of whether the target user requires an image of avatar size or a unique aspect ratio, or a high-definition video clip of tens of seconds, the generation model provided in the embodiments of this application can adaptively generate the content. In addition, in some embodiments, the target user can also flexibly specify output parameters through natural language prompts (such as visual prompt text that can also include specified output parameters), making the generated content more in line with the diverse needs of actual application scenarios.

[0119] Furthermore, this application's embodiments use autoregression to generate only low-resolution latent space feature maps (which can be understood as 'structural sketches'), while delegating high-resolution details to flow matching / diffusion for parallel refinement. In this way, the computational cost of autoregression can be reduced to a very small fraction of its original value (e.g., downsampling by 4 to 8 times, reducing the number of pixels involved in autoregression to approximately 1 / 16–1 / 64 of that in a clear image), while the diffusion part is easier to parallelize. In summary, compared to pure high-resolution autoregression, the method employed in this application's embodiments significantly reduces latency while maintaining global structural consistency and detail quality.

[0120] Reference Figure 4 In some embodiments, the training process of the generative model includes, but is not limited to, steps S401 to S403.

[0121] Step S401: Obtain multimodal training samples of the sample pets, including sample pet images and sample pet videos;

[0122] Step S402: Perform image pre-training on the visual processing sub-model based on sample pet images;

[0123] Step S403: After the image pre-training is completed, the visual processing sub-model is jointly trained based on the sample pet images and sample pet videos.

[0124] In step S401 of some embodiments, multimodal training samples of sample pets can be obtained from multiple pet media databases. The multimodal training samples may include historical pet images (i.e., sample pet images) and historical pet videos (i.e., sample pet videos). These sample pet images and videos may contain information such as the sample pets' diverse postures, environment, and behaviors.

[0125] In steps S402 to S403 of some embodiments, the visual processing sub-model is first pre-trained based on sample pet images. This means the parameters of the visual processing sub-model can be adjusted based on its processing loss on the sample pet images. Then, after sufficient image pre-training (e.g., the training rounds reach a preset number, or the processing loss is less than a preset threshold), the visual processing sub-model is retrained based on sample pet images and sample pet videos. This enables joint training of the visual processing sub-model with both video and images (i.e., joint visual training). It is understood that the data fusion sub-model (including the text encoder and autoregressive encoder) can be a pre-trained model. Therefore, in this embodiment, only the visual processing sub-model (including the image encoder, autoregressive visual decoder, and stream matching decoder) can be trained to give it better visual data processing capabilities. Thus, the parameters of the data fusion sub-model can be frozen, meaning that the parameters of the data fusion sub-model are not adjusted during subsequent training.

[0126] This application describes a method for training a visual processing sub-model based on multimodal data of pet samples collected daily. This enables the visual processing sub-model to accurately capture the unique appearance features, patterns, postures, and behavioral details of the pet. In practical applications, this improves the similarity between the visual data generated by the visual processing sub-model (such as target visual data) and the target pet in terms of appearance and other characteristics. Therefore, this training method can improve the accuracy of the visual processing sub-model in generating visual data of the target pet.

[0127] Secondly, the method of performing image pre-training followed by visual joint training in this application allows the visual processing sub-model to master static image generation capabilities during the image pre-training stage, laying the foundation for learning the temporal dimension changes of dynamic videos in the subsequent visual joint training stage. Furthermore, the visual joint training method ensures that the visual processing sub-model does not forget the semantics and details of static images while learning dynamic video generation, thereby improving the accuracy of visual data generation.

[0128] Furthermore, as described above, this embodiment employs a unified generative model framework that supports the generation of both static images and dynamic videos. The generative model can treat static images as special video frame sequences, uniformly processing image and video tasks to ensure consistency in style and detail. By sharing model parameters, joint training using image and video data is effectively achieved, guaranteeing not only the detail expressiveness of individual frames but also enhancing the continuity and smoothness of video content.

[0129] Understandably, to improve the generation effect, the reference visual data for the target pet typically contains only a single pet. Therefore, the final generated target pet image and / or target pet video also contains only a single pet. When it is desired that the final generated target pet image and / or target pet video contains multiple pets, the user can upload multiple reference visual data (each reference visual data still contains only a single pet). In scenarios where multiple reference visual data are obtained, all reference visual data can be processed based on the same image encoder.

[0130] The specific processes of image pre-training and visual joint training are explained below.

[0131] First, image pre-training will be explained. In some embodiments, the sample pet images include a first pet image and a second pet image, where the resolution of the first pet image is smaller than that of the second pet image. Step S403 may include, but is not limited to, the following sub-steps:

[0132] Noise processing is applied to the first and second pet images;

[0133] The visual processing sub-model is initially trained using the first pet image after noise processing.

[0134] After the initial image training is completed, a hybrid image set is constructed based on the first pet image after noise processing and the second pet image after noise processing;

[0135] The visual processing sub-model is trained at mixed scales based on a mixed image set.

[0136] In this embodiment, image pre-training can be performed in an unsupervised manner. That is, instead of using manually labeled information, the visual processing sub-model learns distribution features from a large amount of image data through generation tasks. Specifically, a flow matching diffusion training framework can be used: noise is randomly added to the acquired training images (i.e., the first pet image and the second pet image), enabling the visual processing sub-model to learn the ability to "remove" noise and restore the image during training (essentially predicting the "velocity vector" from the noise state to the original data state, as defined by flow matching technology). The image pre-training stage includes an initial stage and a mixed-scale training stage. In the initial stage, a large number of low-resolution images (i.e., the first pet image with added noise, with a resolution of 256 pixels, without specific resolution limitations) can be used to pre-train the visual processing sub-model, allowing it to initially learn the basic patterns and low-frequency features of image generation. These basic patterns and low-frequency features refer to the overall structure and low-frequency content of the image, such as the global composition layout, main color blocks, and the approximate shape of objects. These attributes are more pronounced during low-resolution training. The approach of prioritizing the learning of macroscopic patterns and overall contours of an image by the visual processing sub-model can lay the foundation for the generation of subsequent details (high-frequency features).

[0137] After the initial image training is complete (e.g., when the preset number of training epochs is reached), a mixed-scale training phase (e.g., mixing 256-pixel and 512-pixel images) can be performed. Specifically, a mixed image set can be constructed based on a low-resolution first pet image (e.g., 256 pixels) and a high-resolution second pet image (e.g., 512 pixels), and the visual processing sub-model can be trained based on this mixed image set. In this way, the visual processing sub-model can further master high-resolution detail representation while maintaining its low-resolution generation capability.

[0138] In some embodiments, the visual processing sub-model is initially trained based on a noise-processed first pet image, including but not limited to the following sub-steps:

[0139] The visual processing sub-model is invoked to perform denoising processing on the first pet image after noise processing to obtain the predicted noise speed;

[0140] The parameters of the visual processing sub-model are adjusted based on the predicted noise velocity and the actual noise velocity of the first pet image after noise processing.

[0141] In this embodiment, a first pet image with added noise is input into an image encoder, and the processing result obtained based on the image encoder and the autoregressive encoder (such as the noisy visual latent vector corresponding to the first pet image) is used as input data for the visual processing network 301. The visual processing network 301 estimates the noise change rate and direction based on this input data to obtain the predicted noise velocity. The predicted noise velocity can be used to guide the noise vector to move towards the real image direction, that is, it can reflect the visual processing sub-model's estimation of "how noise evolves" into an image. In other words, the visual processing sub-model (such as a stream matching decoder) does not directly output the pixel values ​​of the final image or the noise itself, but outputs an instantaneous velocity vector. This velocity vector is used to describe how to gradually adjust the current sample to remove noise, thereby approximating the real data distribution.

[0142] The prediction loss (i.e., the processing loss mentioned earlier) of the visual processing sub-model can be determined based on the predicted noise rate and the actual noise rate of the corresponding sample. This prediction loss can then be used to adjust the parameters of the visual processing sub-model (including the image encoder, autoregressive visual decoder, and stream matching decoder). Specifically, the mean squared loss between the predicted and actual noise rates can be used as the prediction loss. By minimizing this mean squared loss, the parameters of the visual processing sub-model are iteratively optimized, thereby gradually bringing the final noise removal rate (i.e., the predicted noise rate) output by the visual processing sub-model closer to the ideal value (i.e., the actual noise rate).

[0143] The true noise velocity can refer to the true asymptotic velocity. As a supervisory signal during training, the true asymptotic velocity is calculated using a pre-defined asymptotic path. Specifically, during training, a continuous evolutionary trajectory is defined from random noise samples (such as the first pet image with added noise) to the target real image (such as the original first pet image), used to characterize the process of noise gradually transforming into an image. With this trajectory, the ideal velocity vector, i.e., the true asymptotic velocity, can be calculated at any intermediate time. Furthermore, methods such as linear interpolation paths can be used to determine the true asymptotic velocity; no specific limitations are imposed on this.

[0144] In this embodiment, the predicted noise velocity is compared with the true asymptotic velocity to calculate the mean squared error loss, and the model parameters are adjusted based on the mean squared error loss. In this way, by having the visual processing sub-model repeatedly match the reference velocity field leading from the noise samples to the real image, the visual processing sub-model gradually learns to approximate these true asymptotic velocities, thereby correctly guiding any initial noise towards the true data distribution during inference (e.g., generating high-resolution visual data based on low-resolution visual data).

[0145] It is understandable that the above-mentioned method for calculating loss (i.e., calculating the mean square error between the predicted noise velocity and the actual noise velocity) is basically the same in the initial stage of image pre-training and the mixed-scale training stage, that is, both are for the unsupervised target of reconstructing the image itself.

[0146] As described above, this application integrates autoregressive generation and flow matching (diffusion) generation techniques to achieve a high-quality and highly controllable generation process. By progressively refining the generated content through autoregression and incorporating flow matching diffusion for progressive noise reduction optimization at each generation step, the clarity and temporal consistency of generated images and videos are effectively improved, reducing the error accumulation problem of traditional autoregressive models. Furthermore, this application allows users or developers to dynamically intervene and adjust the generation process (e.g., real-time adjustments based on user feedback on intermediate results or the final output during generation, such as adjusting visual cue text, locally redrawing the target pet image and / or target pet video, etc., without retraining the model), thereby more accurately ensuring that the generated results meet user expectations.

[0147] Secondly, the visual joint training is explained. In some embodiments, the sample pet videos include a first pet video, a second pet video, and a third pet video. The first pet video is a pet video with a resolution within a preset first resolution range and a video duration within a preset first duration range. The second pet video is a pet video with a resolution within the first resolution range and a video duration within a preset second duration range, where the lower limit of the second duration range is greater than the upper limit of the first duration range. The third pet video is a pet video with a resolution within the preset second resolution range and a video duration within the second duration range, where the lower limit of the second resolution range is greater than the upper limit of the first resolution range.

[0148] Visual joint training includes a first-stage training, a second-stage training, and a third-stage training. Step S404 may include, but is not limited to, the following sub-steps:

[0149] The visual processing sub-model is trained in the first stage based on sample pet images and the first pet video.

[0150] The visual processing sub-model is trained in the second stage based on sample pet images and second pet videos.

[0151] The visual processing sub-model is trained in the third stage based on sample pet images and third-party pet videos.

[0152] In this embodiment, the sample pet videos include a first pet video, a second pet video, and a third pet video. These three types of pet videos are distinguished based on preset first resolution range, second resolution range, first duration range, and second duration range. For example, the first resolution range is (A, B), the second resolution range is (C, D), the first duration range is (T1, T2), and the second duration range is (T3, T4). The value of B is less than the value of C, meaning the first resolution range is a "low resolution range," and the second resolution range is a "high resolution range." The value of T2 is less than the value of T3, meaning the first duration range is a "short duration range," and the second duration range is a "long duration range." Thus, the first pet video can refer to a "low-resolution short video," the second pet video can refer to a "low-resolution long video," and the third pet video can refer to a "high-resolution long video." This embodiment does not specifically limit the specific values ​​of the first resolution range, second resolution range, first duration range, and second duration range.

[0153] The visual joint training can be divided into three stages. The first stage trains the visual processing sub-model using image-video joint training based on a first pet video (a low-resolution short video) and sample pet images (first pet image and / or second pet image), enabling the visual processing sub-model to learn basic video generation consistency. The second stage trains the visual processing sub-model using image-video joint training based on a second pet video (a low-resolution long video) and sample pet images (first pet image and / or second pet image), enabling the visual processing sub-model to learn long-term dynamic consistency. The third stage trains the visual processing sub-model using image-video joint training based on a third pet video (a high-resolution long video) and sample pet images (first pet image and / or second pet image), enabling the visual processing sub-model to improve the detail clarity of the generated results while maintaining long-term coherence. It can be understood that image-video joint training refers to constructing a training set based on a parallel mixture of images and videos, so that each training batch contains both video frame sequences and static images. This design ensures that the visual processing sub-model does not forget the semantics and details of static images while learning dynamic video generation. Furthermore, the design of the joint image-video training loss function is the same as that for image pre-training. That is, the same generation loss (such as the mean squared loss between the predicted noise velocity and the actual noise velocity under the flow matching diffusion framework, i.e., the aforementioned prediction loss) is used to train the visual processing sub-model for both image and video samples. For video data, the visual processing sub-model needs to predict the denoising result of a noisy video frame sequence. For image data, the visual processing sub-model needs to predict the denoising result of a noisy image. Since an image can be considered a single frame of video, image data and video data have the same training objective. Based on this, this embodiment does not set different losses for image data and video data, but rather optimizes the same loss function uniformly, only distinguishing between image and video sources. In practical applications, the sampling weights of the corresponding data can be dynamically adjusted according to the data source as needed to ensure a balance between the contributions of video data and image data to the training. (For example, if there is limited image and video data, higher sampling weights can be assigned to video data and lower weights to image data. This way, in each training batch, the model has a more similar chance of seeing videos and images, preventing one type of data from dominating the training process, thereby improving the model's generalization ability and balance.) Overall, however, the loss function in the visual joint training phase is the same as that in the image pre-training phase. The visual processing sub-model processes both static frames and video frame sequences simultaneously, enabling it to learn to generate both high-quality single-frame images and multi-frame continuous videos within the same framework.

[0154] This application's embodiments, based on image pre-training, further introduce video data (i.e., sample pet videos) and gradually increase the duration and resolution of video generation (from short-duration, low-resolution videos to long-duration, high-resolution videos). Through image-video joint training, the visual processing sub-model adapts to both image and video generation tasks, ensuring high-quality and stable generated content. Secondly, dividing the visual joint training into three stages gradually increases the difficulty of the video generation task, allowing the visual processing sub-model to gradually adapt to generation tasks with longer time spans and higher resolutions.

[0155] It is understood that in this embodiment, image-video joint training is initiated only after sufficient image pre-training is completed. That is, video data is introduced to begin joint training only after the visual processing sub-model has been pre-trained to a certain extent on large-scale image data and has mastered the ability to generate static images. Specifically, the aforementioned two stages of image pre-training are first performed (including initial image training using 256-pixel images and mixed-scale training using 256-pixel and 512-pixel images) until the image generation performance of the visual processing sub-model converges, and its consistency and detail reach a good level. This level is typically manifested in the visual processing sub-model being able to generate clear and semantically correct images with low loss and possessing a certain multi-scale generation capability. Reaching this state indicates that the visual processing sub-model has learned rich static visual features, laying the foundation for learning changes in the temporal dimension, at which point video data is incorporated into the training.

[0156] It's understandable that, for the three stages of joint visual training, the model parameters obtained from the previous stage can be used as the initial model parameters for the next stage, ensuring that the visual processing sub-model gradually masters more complex temporal and spatial information. However, this doesn't mean that the next stage must be started only after the previous stage has fully converged. In actual training, data from different stages can be mixed during training through methods such as data bucketing and random sampling. For example, samples can be randomly drawn from data buckets with different video lengths or resolutions during training, so that the visual processing sub-model can be exposed to videos of various lengths and / or resolutions in each training step. This approach preserves the learning progress from easy to difficult while preventing the visual processing sub-model from forgetting the knowledge learned in previous stages, thus improving training robustness. Therefore, the three stages can be executed sequentially, but in practice, they can also be appropriately parallelized or interleaved in sampling to maintain training flexibility and efficiency.

[0157] In some embodiments, the training process of the generative model may include text-visual joint training in addition to the image pre-training stage and the vision joint training stage described above. The text-visual joint training is described below.

[0158] The training process for a generative model may also include, but is not limited to, the following steps:

[0159] Get sample prompt text;

[0160] Construct training sample pairs based on sample pet images and sample prompt text, or sample pet videos and sample prompt text;

[0161] After the visual joint training is completed, the data fusion sub-model and the visual processing sub-model are jointly trained as a whole based on the training samples.

[0162] In this embodiment, as described above, the text encoder and autoregressive encoder can directly use pre-trained open-source models, so the joint text-vision training phase is not mandatory. However, sample pet images (or sample pet videos) collected by smart devices can be combined with sample prompt text to form training sample pairs, and the overall generative model can be trained based on these training sample pairs. This provides the generative model with more realistic scenarios from target users, enabling it to generate more pet images and / or videos that closely resemble real-world scenarios in practical applications. Specifically, the feedback from target users in pet management applications can be used as sample prompt text, or sample prompt text can be generated based on a VLM (Vision-Language Model), etc., without specific limitations. The function of sample prompt text is similar to that of visual prompt text; that is, sample prompt text refers to text used to describe the content (such as scenes, actions, effects, etc.) corresponding to the expected final generated visual data (i.e., images or videos). In the joint text-vision training phase, the input data consists of sample pet images (or sample pet videos) and corresponding natural language descriptions (i.e., sample prompt text). Thus, the sample prompt text is converted into a corresponding text vector by a text encoder, while the sample pet image (or sample pet video) is used to extract its corresponding image vector by an image encoder. These text and image vectors are then processed sequentially by an autoregressive encoder, an autoregressive visual decoder, and a stream matching decoder, ultimately outputting the corresponding predicted noise velocity. Based on this predicted noise velocity and the aforementioned loss definition, the prediction loss can be calculated. In other words, during the joint text-visual training phase, the mean squared error of velocity prediction within the stream matching framework is still used as the loss.

[0163] Understandably, since the text-visual joint training phase trains the entire generative model, the parameters of the text encoder and autoregressive encoder can be unfrozen (unlike the image pre-training and vision joint training phases where the text encoder and autoregressive encoder are in a parameter-frozen state). Thus, the parameters of the text encoder, image encoder, autoregressive encoder, autoregressive visual decoder, and stream matching decoder can be adjusted based on the current prediction loss. This method of jointly training the entire generative model ensures that the text prompts, low-level visual structures, and final output data are adjusted in an integrated manner during the text-visual joint training, enabling the generative model to simultaneously improve semantic understanding and image (or video) synthesis quality.

[0164] Furthermore, to enhance the consistency between text and vision, a cross-modal alignment term or contrastive loss can be added to the loss function of joint text-visual training. This can bring matching text-visual feature pairs (i.e., sample cue text-sample pet image, or sample cue text-sample pet video) closer together, while pushing away mismatched text-visual feature pairs, thereby maximizing the similarity between text and generated visual features. For example, matching text-visual feature pairs have high similarity, while mismatched text-visual feature pairs have low similarity. The loss function encourages matching pairs to have much higher similarity than mismatched pairs, thus mapping semantically consistent text and visual content to similar regions in the representation space. When the input data is video data, a temporal consistency loss can also be added (e.g., encouraging smooth features in adjacent frames or continuous optical flow) to improve temporal coherence. However, it should be noted that regardless of whether additional task losses are added, the core stream matching mean squared error generation loss is always used throughout image and video training.

[0165] Understandably, after completing image pre-training, image-video joint training, and text-visual joint training (optional), the generative model can be fine-tuned. Specifically, the generative model can be fine-tuned using pet data specific to the target user to further refine the model parameters, accurately capture the individual characteristics of the target pet, ensure that the generated content highly matches the actual pet, and enable the model to have personalized visual data generation capabilities.

[0166] The pet visual data generation method provided in this application trains a generative model using historical pet images and / or videos collected by a smart device, along with natural language prompts input by the target user, thereby achieving highly personalized pet image or video generation. By employing a unified generative model framework, it simultaneously supports the generation of static images and dynamic videos. Through autoregressive block generation and stream matching techniques, it achieves high flexibility in output size and duration.

[0167] Furthermore, in some embodiments, the embodiments of this application can also be optimized for cloud and edge devices, supporting an efficient edge-cloud collaborative architecture. The generation module can employ inference acceleration strategies, such as block-parallel generation and model distillation techniques, to significantly reduce generation latency. Block-parallel generation refers to utilizing the characteristics of autoregressive block generation to simultaneously predict multiple elements within a block, thereby reducing sequence steps and increasing parallelism. This method can effectively reduce inference latency. Model distillation refers to using a teacher model to guide a student model, enabling the student model to achieve similar generation results to the teacher model with a smaller scale or fewer steps, thereby compressing the model and accelerating inference. When the above two methods are combined, the generation model can significantly improve generation efficiency (such as significantly shortening inference time) while ensuring generation quality.

[0168] In one specific embodiment, the pet visual data generation method provided in this application may include the following steps:

[0169] 1. The target user sends a request to generate a pet image (or video) through the AI ​​assistant interface within the mobile pet management application. Specifically, the target user can input natural language prompts (i.e., visual prompt text) in the AI ​​assistant interface. These prompts describe the desired pet scene, action, or effect, such as "Generate a sci-fi style pet profile picture based on my pet XX's reference image" or "Generate a video of my pet cat playing in the garden."

[0170] 2. The AI ​​assistant can process the above-mentioned generation requests: The AI ​​assistant can understand the target user's request intent based on a Large Language Model (LLM), and after recognizing that the request intent is to generate pet content, it calls the visual data generation service (i.e., calls the generation model) through a predefined tool interface. It sends the prompt text and prompt image input by the target user, as well as the pet reference image obtained from the pet media database based on the pet's identity information, to the generation model.

[0171] 3. The pre-trained generative model is invoked to generate images (or videos) based on the input data. After the generative model outputs images or consecutive video frames, the generated content can be decoded and post-processed to generate the final visualized pet image or video file, which is then returned to the target user through the AI ​​assistant interface.

[0172] 4. The target user can provide feedback on the generated content, such as requesting modifications or offering more detailed suggestions. After receiving the feedback data from the target user, the generation strategy can be dynamically adjusted based on the feedback data, and the adjusted generated content can be returned to the target user.

[0173] Reference Figure 5In some embodiments, this application also provides a pet visual data generation device 500, which includes:

[0174] The request acquisition unit 510 is used to receive a data generation request, which includes visual cue text and reference visual data of the target pet.

[0175] The model invocation unit 520 is used to invoke a pre-trained generative model based on the visual cue text and reference visual data. The generative model includes a data fusion sub-model and a visual processing sub-model.

[0176] The data fusion unit 530 is used to call the data fusion sub-model to perform data fusion based on visual cue text and reference visual data to obtain a visual latent vector.

[0177] The visual processing unit 540 is used to call the visual processing sub-model to generate visual data based on the visual latent vector, and obtain target visual data corresponding to the visual prompt text. The target visual data includes target pet images and / or target pet videos.

[0178] Model training unit 550, used for training the generative model, includes:

[0179] The sample acquisition subunit is used to acquire multimodal training samples of sample pets, including sample pet images and sample pet videos;

[0180] The image pre-training subunit is used to pre-train the visual processing sub-model based on sample pet images.

[0181] The visual joint training subunit is used to perform visual joint training on the visual processing submodel based on sample pet images and sample pet videos after image pre-training.

[0182] Reference Figure 6 , Figure 6 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes:

[0183] The processor 601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0184] The memory 602 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called and executed by the processor 601 using the pet visual data generation method of the embodiments of this application.

[0185] The input / output interface 603 is used to implement information input and output;

[0186] The communication interface 604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0187] Bus 605 transmits information between various components of the device (e.g., processor 601, memory 602, input / output interface 603, and communication interface 604);

[0188] The processor 601, memory 602, input / output interface 603, and communication interface 604 are connected to each other within the device via bus 605.

[0189] This application also provides a storage medium storing a computer program, which, when executed by a processor, implements the pet visual data generation method provided in this application.

[0190] This application also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the above-described pet visual data generation method.

[0191] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0192] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0193] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0194] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0195] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0197] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0199] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A pet visual data generation method, characterized by, The method comprises: receiving a data generation request, the data generation request comprising visual prompt text and reference visual data of a target pet; wherein the reference visual data is determined from a pet media database; calling a pre-trained generation model according to the data generation request, wherein the generation model comprises a data fusion sub-model and a visual processing sub-model, the data fusion sub-model comprises a text encoder and an autoregressive encoder, and the visual processing sub-model comprises an image encoder and a visual processing network; the visual processing network comprises an autoregressive visual decoder and a flow matching decoder; calling the image encoder to perform visual feature extraction on the reference visual data to obtain a target image vector; calling the data fusion sub-model to perform data fusion based on the visual prompt text and the reference visual data to obtain a visual latent vector, comprising: calling the text encoder to perform semantic feature extraction on the visual prompt text to obtain a target text vector; calling the autoregressive encoder to perform attention fusion on the target text vector and the target image vector to obtain the visual latent vector; calling the visual processing sub-model to generate visual data based on the visual latent vector to obtain target visual data corresponding to the visual prompt text, the target visual data comprising a target pet image and / or a target pet video, comprising: calling the autoregressive visual decoder to decode the visual latent vector to obtain preliminary visual data; calling the flow matching decoder to perform denoising processing on the preliminary visual data to obtain the target visual data, wherein the resolution of the target visual data is greater than that of the preliminary visual data; wherein the training process of the generation model comprises: obtaining multi-modal training samples of a sample pet, the multi-modal training samples comprising sample pet images and sample pet videos; performing image pre-training on the visual processing sub-model based on the sample pet images; after image pre-training is completed, performing visual joint training on the visual processing sub-model based on the sample pet images and the sample pet videos, wherein the visual joint training comprises three stages of image-video joint training, image-video joint training refers to constructing a training set based on parallel mixing of images and videos, and the three stages gradually increase the length and resolution of video generation.

2. The method of claim 1, wherein, The sample pet images comprise first pet images and second pet images, and the resolution of the first pet images is less than that of the second pet images. The image pre-training on the visual processing sub-model based on the sample pet images comprises: performing noise processing on the first pet images and the second pet images; performing initial image training on the visual processing sub-model based on the first pet images after noise processing; after initial image training is completed, constructing a mixed image set according to the first pet images after noise processing and the second pet images after noise processing; performing mixed scale training on the visual processing sub-model based on the mixed image set.

3. The method of claim 2, wherein, The first pet image after noise processing is used for initial image training of the visual processing sub-model, including: The visual processing sub-model is called to perform denoising processing on the first pet image after noise processing to obtain a predicted noise speed; The visual processing sub-model is adjusted in parameters according to the predicted noise speed and a real noise speed of the first pet image after noise processing.

4. The method of claim 1, wherein, The sample pet video includes a first pet video, a second pet video and a third pet video, the first pet video is a pet video with a resolution in a preset first resolution range and a video length in a preset first length range, the second pet video is a pet video with a resolution in the first resolution range and a video length in a preset second length range, a lower limit of a value range of the second length range is greater than an upper limit of a value range of the first length range, and the third pet video is a pet video with a resolution in a preset second resolution range and a video length in the second length range, a lower limit of a value range of the second resolution range is greater than an upper limit of a value range of the first resolution range; The visual joint training includes first stage training, second stage training and third stage training; The visual joint training of the visual processing sub-model based on the sample pet image and the sample pet video includes: The first stage training of the visual processing sub-model based on the sample pet image and the first pet video; The second stage training of the visual processing sub-model based on the sample pet image and the second pet video; The third stage training of the visual processing sub-model based on the sample pet image and the third pet video.

5. The method of claim 1, wherein, The training process of the generation model further includes: Obtaining sample prompt text; According to the sample pet image and the sample prompt text, or the sample pet video and the sample prompt text, a training sample pair is constructed; After the visual joint training is completed, the data fusion sub-model and the visual processing sub-model are jointly trained as a whole based on the training sample pair.

6. The method of claim 1, wherein, The receiving of the data generation request includes: Receiving the visual prompt text and pet identity information of a target pet; According to the pet identity information, historical visual data is extracted from a corresponding pet media database; In response to a selection operation on the historical visual data, the reference visual data is determined.

7. A pet visual data generating apparatus, characterized by comprising: The apparatus includes: A request obtaining unit configured to receive a data generation request, the data generation request including a visual prompt text and reference visual data of a target pet; wherein the reference visual data is determined from a pet media database; A model calling unit configured to call a pre-trained generation model according to the data generation request, wherein the generation model includes a data fusion sub-model and a visual processing sub-model, the data fusion sub-model includes a text encoder and a self-recurrent encoder, and the visual processing sub-model includes an image encoder and a visual processing network; the visual processing network includes a self-recurrent visual decoder and a flow matching decoder; calling the image encoder to perform visual feature extraction on the reference visual data to obtain a target image vector; the data fusion unit is configured to call the data fusion sub-model to perform data fusion on the visual prompt text and the reference visual data to obtain a visual latent vector, including: calling the text encoder to perform semantic feature extraction on the visual prompt text to obtain a target text vector; and calling the autoregressive encoder to perform attention fusion on the target text vector and the target image vector to obtain the visual latent vector; the visual processing unit is configured to call the visual processing sub-model to generate target visual data corresponding to the visual prompt text based on the visual latent vector, the target visual data including a target pet image and / or a target pet video, including: calling the autoregressive visual decoder to decode the visual latent vector to obtain preliminary visual data; and calling the flow matching decoder to perform denoising processing on the preliminary visual data to obtain the target visual data, wherein the resolution of the target visual data is greater than that of the preliminary visual data. the model training unit is configured to train the generation model, including: the sample acquisition sub-unit is configured to acquire a multi-modal training sample of a sample pet, the multi-modal training sample including a sample pet image and a sample pet video; the image pre-training sub-unit is configured to pre-train the visual processing sub-model based on the sample pet image; the visual joint training sub-unit is configured to, after the image pre-training is completed, perform visual joint training on the visual processing sub-model based on the sample pet image and the sample pet video, wherein the visual joint training includes image-video joint training in three stages, the image-video joint training referring to constructing a training set based on parallel mixing of images and videos, and the three stages gradually increasing the length and resolution of video generation.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the pet visual data generation method in any one of claims 1-6.

9. A computer readable storage medium, the storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the pet visual data generation method in any one of claims 1-6.

10. A computer program product, characterised in that, The computer program product includes a computer program that is read and executed by a processor of a computer device, so that the computer device executes the pet visual data generation method in any one of claims 1-6.

Images