Method and device for training model, controller, refrigeration equipment and product

Abstract

The embodiment of the invention relates to a method and device for training a model, a controller, refrigeration equipment and a computer program product. The method includes determining, by a task processing model, a predicted tag based on the first feature and the second feature. The method further includes generating a reconstructed image based on the first feature. The method further includes training a first encoder, a first decoder, and a task processing model according to the prediction tag, the reconstructed image, the first sample, and the sample tag. According to the method disclosed by the embodiment of the invention, the first encoder can be supervised by using the reconstructed image in the training process, and the first encoder is prevented from losing excessive image information during dimension alignment, so that the task processing model can complete the prediction task based on the image information related to the prediction task; therefore, the prediction accuracy is improved.

Description

Technical Field

[0001] This disclosure relates to the field of machine learning, and more specifically to methods, apparatus, controllers, cooling devices, and computer program products for training models. Background Technology

[0002] Machine learning technology has made significant progress in recent years. By building algorithmic models, it enables computers to learn from data and make predictions or decisions. Neural network-based models, such as convolutional neural networks and recurrent neural networks, have achieved breakthroughs in many fields, including image recognition and natural language processing. Technological advancements have made tasks such as image analysis, recognition, and generation more efficient and accurate, playing a crucial role in areas such as medical image diagnosis, autonomous driving, and security monitoring.

[0003] For example, machine learning models can be used to analyze data sensed by various sensors within refrigeration equipment, thereby predicting the state of objects (such as food) inside the equipment. This eliminates the need for people to constantly turn on refrigeration equipment and guess the state of objects inside based on subjective feelings. These models make people's lives more convenient. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, controller, cooling device, and computer program product for training a model. In a first aspect of this disclosure, a method for training a model is provided. The method includes determining a predicted label using a task processing model based on a first feature and a second feature, wherein the first feature is determined by a first encoder based on a first sample belonging to an image modality, and the second feature is determined by a second encoder based on a second sample belonging to a non-image modality. The method also includes generating a reconstructed image by a first decoder based on the first feature. The method further includes training a first encoder, a first decoder, and a task processing model based on the predicted label, the reconstructed image, the first sample, and the sample label.

[0005] In a second aspect of this disclosure, a method for data processing is provided. The method includes acquiring first data and second data, wherein the first data belongs to an image modality and the second data belongs to a non-image modality. The method further includes determining a target prediction value based on the first and second data using a first encoder, a second encoder, and a trained task processing model according to the first aspect of this disclosure.

[0006] In a third aspect of this disclosure, an apparatus for training a model is provided. The apparatus includes a first prediction unit configured to determine a predicted label using a task processing model based on a first feature and a second feature, wherein the first feature is determined by a first encoder based on a first sample belonging to an image modality, and the second feature is determined by a second encoder based on a second sample belonging to a non-image modality. The apparatus also includes a reconstruction unit configured to generate a reconstructed image by a first decoder based on the first feature. The apparatus further includes a training unit configured to train the first encoder, the first decoder, and the task processing model based on the predicted label, the reconstructed image, the first sample, and the sample label.

[0007] In a fourth aspect of this disclosure, an apparatus for processing data is provided. The apparatus includes an acquisition unit configured to acquire first data and second data, wherein the first data belongs to an image modality and the second data belongs to a non-image modality. The apparatus also includes a second prediction unit configured to, based on the first and second data, use a first encoder, a second encoder, and a trained task processing model according to a first aspect of this disclosure to determine a target prediction value.

[0008] In a fifth aspect of this disclosure, a controller is provided, comprising: at least one processor; and a memory coupled to the at least one processor and having instructions stored thereon, the instructions causing the controller to perform a method according to a first or second aspect of this disclosure when executed by the at least one processor.

[0009] In a sixth aspect of this disclosure, a refrigeration apparatus is provided, including a controller according to a fifth aspect of this disclosure.

[0010] In a seventh aspect of this disclosure, a computer program product is provided, which is tangibly stored on a non-transient computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to perform a method according to a first or second aspect of this disclosure.

[0011] In an eighth aspect of this disclosure, a computer-readable storage medium is provided that stores machine-executable instructions thereon, wherein the machine-executable instructions are executed by a processor to implement the method according to a first or second aspect of this disclosure.

[0012] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0014] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure may be implemented is shown;

[0015] Figure 2 A flowchart of a method for training a model according to an embodiment of the present disclosure is shown;

[0016] Figure 3 A diagram illustrating the training process in a scenario of predicting the freshness of food according to an embodiment of the present disclosure is shown.

[0017] Figure 4 A schematic diagram illustrating the determination of training loss according to an embodiment of the present disclosure is shown;

[0018] Figure 5 A schematic diagram of an image encoder structure according to an embodiment of the present disclosure is shown;

[0019] Figure 6 A schematic diagram illustrating an application scenario according to an embodiment of the present disclosure is shown;

[0020] Figure 7 A schematic diagram of a method for training a model according to an embodiment of the present disclosure is shown;

[0021] Figure 8 A schematic diagram of a method for model training according to another embodiment of the present disclosure is shown;

[0022] Figure 9A A process diagram illustrating a method for data processing according to an embodiment of the present disclosure is shown;

[0023] Figure 9B A flowchart of a method for data processing according to an embodiment of the present disclosure is shown;

[0024] Figure 10 This is a schematic diagram of an apparatus for training a model according to an embodiment of the present disclosure;

[0025] Figure 11 This is a schematic diagram of an apparatus for processing data according to embodiments of the present disclosure; and

[0026] Figure 12 A schematic block diagram of an example device suitable for implementing embodiments of the present disclosure is shown.

[0027] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0028] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0029] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0030] In many fields, machine learning models are needed to handle multimodal samples. Samples from different modalities may have different dimensions. Using samples from multiple modalities for prediction can fully utilize information from different perspectives related to the prediction task, thus enabling accurate predictions. For example, a cross-attention module can receive two samples and combine their features to obtain fused features relevant to the prediction task. Cross-attention modules typically require the input features to have the same dimension; otherwise, correlation matrix operations cannot be performed within the module. Other deep learning modules also require prediction based on features with similar dimensions.

[0031] If one modality is an image modality and the other is a non-image modality, such as a numerical modality, the dimensionality difference between the samples of these two modalities is often very large in a computer. For example, an image modality sample might be represented as a 1920×1080 matrix (depending on the sample resolution), where each element represents a pixel value, while a non-image modality sample might be represented as a 7×1 matrix, where each element represents a sensed value. Related techniques often employ methods that compress only the dimensions of the image modality samples to achieve dimensionality alignment, but this results in the loss of image information, leading to a decrease in prediction accuracy.

[0032] To address this, this disclosure proposes a method for training a model. This method trains the encoder's ability to retain image information and the decoder's ability to reconstruct image information in parallel during task processing model training, and trains the model based on image loss and prediction loss; that is, it uses image loss to supervise the training of the task processing model. According to the method of this disclosure, the first encoder can be supervised using reconstructed images during training, preventing the first encoder from losing too much image information during the alignment process. This allows the task processing model to complete the prediction task based on image information relevant to the prediction task, thereby improving prediction accuracy.

[0033] The embodiments of this disclosure will now be described in further detail with reference to the accompanying drawings, wherein... Figure 1 This is a schematic diagram of an example environment 100 in which various embodiments of this disclosure may be implemented. For example... Figure 1 As shown, the example environment 100 includes a first encoder 106, a second encoder 108, and a task processing model 114. In this embodiment, at least the first encoder 106, the first decoder 110, and the task processing model 114 are the training objects.

[0034] Based on the first feature e1 and the second feature e2, the predicted label 116 is determined by the task processing model 114. The first feature e1 is determined by the first encoder 106 based on a first sample 102 belonging to the image modality, and the second feature e2 is determined by the second encoder 108 based on a second sample 104 belonging to a non-image modality. The first feature e1 and the second feature e2 have similar or identical dimensions. In this embodiment, the first sample can be represented by a first representation 102, and the second sample can be represented by a second representation 104. Figure 1 The second representation 104 is a one-dimensional vector, which is merely an example and does not constitute a limitation of this disclosure. In this embodiment, the dimension of the first representation 102 is typically much larger than the dimension of the second representation 104. If the first representation 102 is directly compressed to a dimension similar to or the same as that of the exemplary second representation 104 (e.g., one-dimensional), a large amount of image information may be lost, leading to inaccurate predictions. The first encoder 106 and the second encoder 108 may have similar model structures.

[0035] Based on the first feature e1, the first decoder 110 generates a reconstructed image 112. Since the first sample is an image modality sample, the subsequent prediction process should theoretically be based on image information, rather than on other information unrelated to the image. Therefore, to ensure that the first encoder 106 can primarily retain image information during training, the reconstructed image 112 can be generated based on the first feature e1. In some embodiments, the reconstructed image 112 obtained after training should at least contain image information that sufficiently reflects information relevant to the prediction task. The first decoder 110 can be used to decode the first feature e1 to obtain the reconstructed image 112.

[0036] Based on the predicted label 116, the reconstructed image 112, the first sample, and the sample label, a first encoder 106, a first decoder 110, and a task processing model 114 are trained. The sample label refers to the ground truth value corresponding to the first and second samples for the prediction task. In classification tasks, the sample label is a discrete value representing the category to which the first and second samples belong. For example, in a freshness prediction task, the sample label can be "0" or "1," representing two different freshness categories (fresh or rotten), respectively. In regression tasks, the sample label is a continuous value representing the specific numerical value to be predicted. For example, in a freshness prediction task, the sample label indicates the actual freshness (e.g., freshness of 80%). This disclosure includes, but is not limited to, the above-described task categories.

[0037] In this embodiment, the reconstructed image 112 acts as a supervisory factor, comparing it to the first sample to determine the degree of image information loss, thereby training the first encoder 106's ability to retain image information for tasks. Prediction labels 116 and sample labels are used to determine the prediction loss, thereby training the task processing model's prediction ability for the prediction task. This training method ensures that the first encoder 106 retains as much image information as possible while training its prediction ability, making the prediction ability trained based on image information. Therefore, the trained first encoder and task processing model can primarily predict based on image information and have good prediction accuracy.

[0038] exist Figure 1 In the application scenarios shown, the methods according to some embodiments of this application are implemented on a server, but this is merely illustrative and not restrictive. The methods according to some embodiments of this application can also be implemented on other entities with sufficient computing resources and capabilities. Furthermore, the first encoder 106, the second encoder 108, and the task processing model 114 can be located on different entities.

[0039] The above combination Figure 1An example environment 100 in which embodiments of this disclosure can be implemented is described. The following is in conjunction with... Figure 2 A flowchart describing a method 200 for training a model according to an embodiment of the present disclosure is provided. At block 202, a predicted label is determined by a task processing model based on a first feature and a second feature, wherein the first feature is determined by a first encoder based on a first sample belonging to an image modality, and the second feature is determined by a second encoder based on a second sample belonging to a non-image modality.

[0040] The dimensionality of the first sample is often much larger than that of the second sample. Because the task processing model 114 is subsequently used to fuse the first and second samples, it is necessary to align the first and second samples in terms of dimensions to complete the subsequent calculations. In this disclosure, "alignment" means that the dimensions of two or more features are changed to a similar or equal number of dimensions. The task processing model 114 may also include a prediction module for making predictions based on the fused features; this can be any prediction module capable of performing the prediction task, such as a prediction head containing multiple fully connected network layers. Through the first encoder and the second encoder, feature extraction and dimensional alignment of the first and second samples can be achieved simultaneously.

[0041] At box 204, a reconstructed image is generated by the first decoder based on the first feature. The first feature e1 is a feature obtained by encoding the first sample using the first encoder 106 and then extracting and processing its dimensionality. The first decoder 110 has the ability to increase dimensionality. To ensure that the training of subsequent prediction capabilities is based on image information rather than other information, it is necessary to ensure that the first encoder 106 retains as much image information as possible. Here, by generating the reconstructed image through the first decoder 110, we can see how much image information the first feature retains.

[0042] At box 206, a first encoder, a first decoder, and a task processing model are trained based on the predicted label, the reconstructed image, the first sample, and the sample label. In some embodiments, the various sub-models included in the first encoder 106, the second encoder 108, and the task processing model 114 can also be trained together based on the predicted label, the reconstructed image, the first sample, and the sample label. If the second encoder 108 is a pre-trained model, then only the first encoder 106, the first decoder 110, and the task processing model 114 need to be trained.

[0043] In this operation, the reconstructed image 112 acts as a supervisory factor, comparing it to the first sample to determine the degree of loss of image information, thereby training the first encoder 106's ability to preserve image information. Predicted labels 116 and sample labels are used to determine the prediction loss, thereby training the first encoder 106 and the task processing model 114's prediction capabilities for the prediction task. This training method ensures that the first encoder 106 preserves as much image information as possible relevant to the prediction task while aligning dimensions, making the prediction ability trained based on relevant image information. Therefore, the trained first encoder 106 and task processing model 114 can primarily predict based on image information and have good prediction accuracy.

[0044] According to the method disclosed herein, the first encoder can be supervised using reconstructed images during training to prevent the first encoder from losing too much image information during the alignment of dimensions. This allows the task processing model to complete the prediction task based on image information related to the prediction task, which can improve the accuracy of the prediction.

[0045] In some embodiments, the first feature may be determined by a first encoder based on a first representation 102, which can be used to indicate a first sample belonging to an image modality. The second feature may be determined by a second encoder based on a second representation 104, which can be used to indicate a second sample belonging to a non-image modality. The first representation 102 and the second representation 104 have different dimensions. For example, the dimension of the first representation 102 may be 100×100, while the dimension of the second representation 104 may be 1×10. The dimension of the first representation 102 can be much larger than the dimension of the second representation 104. Some task processing models require that the dimensions of two input vectors be exactly the same when performing operations on them. Therefore, in this embodiment, the first feature and the second feature, after alignment using the first encoder and the second encoder, have the same dimension. In this way, the cross-attention module can complete the prediction task by calculating the first feature and the second feature. Optionally, the task processing model includes a cross-attention module that needs to process vectors with the same dimension. In some embodiments, the method further includes: obtaining the first representation 102 based on the first sample and obtaining the second representation 104 based on the second sample. The first representation 102 can represent the distribution of pixels (pixel matrix) in the first sample belonging to the image modality, while the second representation 104 can represent the vector representation, string representation, etc., of the second sample not belonging to the image modality. For the first representation, for example, pixels from the first sample can be cleaned, retaining pixels of reference value, and a pixel matrix can be generated based on the relative positions of the pixels, serving as the first representation. For the second representation, for example, data from the second sample can be arranged in chronological order, and a vector representation can be generated based on the arranged data, serving as the second representation.

[0046] Figure 3A training process diagram is shown in the scenario of predicting the freshness of food according to an embodiment of this disclosure. At 302, image samples (i.e., the first sample) and gas data (e.g., electronic nose data) samples (i.e., the second sample) are received to predict the freshness of food; these two samples have different modalities. The gas data samples are data sensed by a gas sensor. A gas sensor is a sensor that mimics the human olfactory system, capable of detecting, identifying, and analyzing various gases and volatile organic compounds. A gas sensor typically consists of a gas-sensitive sensor array and a corresponding pattern recognition algorithm. The gas-sensitive sensor array is the core component of the gas sensor, composed of multiple chemical or physical sensors with different characteristics, which have different response patterns to different types of gases. The pattern recognition algorithm is used to process the data generated by the sensor array, parsing specific odor information through data analysis and pattern recognition techniques, such as principal component analysis, linear discriminant analysis, support vector machines, artificial neural networks, etc. The image samples and gas data samples are then processed into representations that can be understood by a machine learning model. For example, image samples can be converted into a 1920×1080 pixel matrix, and gas data samples can be converted into a 10×3 matrix.

[0047] At point 304, an image encoder encodes the image samples into a first feature, and a non-image encoder encodes the gas data samples into a second feature. At point 306, the first and second features are input into a cross-attention module. The cross-attention module fuses the first and second features, enhancing or amplifying features related to freshness (e.g., giving them higher attention weights), while weakening features unrelated to freshness (e.g., giving them lower attention weights). This combines the freshness-related features from both the first and second features into a single feature, which helps the prediction module accurately predict freshness. The fused feature is then input into the prediction module for prediction, thus obtaining the predicted label.

[0048] At 308, an image decoder (i.e., the first decoder) is used to generate a reconstructed image based on the first feature. For example, the image decoder may have a structure corresponding to the image encoder, or, for example, the inverse of the image encoder's structure. At 310, a first reconstruction loss and a prediction loss are calculated. The loss function can be any suitable loss function, such as the cross-entropy loss function. The first reconstruction loss is determined based on the difference between the first sample and the reconstructed image, and the prediction loss is determined based on the difference between the sample label and the predicted label. The weighted sum of the first reconstruction loss and the prediction loss can be used as the training loss for this training. In some embodiments, the weights of the first reconstruction loss and the prediction loss can be preset. At 312, the image encoder, the image decoder, and the cross-attention module are trained based on this training loss.

[0049] In this embodiment, by aggregating the first reconstruction loss and the prediction loss, a training loss that simultaneously reflects the image information extraction capability of the image encoder and the prediction capability of the task processing model can be obtained, thereby enabling the training of the prediction capabilities of the image encoder and the cross-attention module. In this embodiment, an odor sensor can be used instead of a gas sensor, and odor data samples can be used instead of gas data samples for prediction. An odor sensor is a measuring device that measures the strength of an odor and represents it digitally. When odorous substances are present in the environment, odor molecules come into contact with the sensing element of the sensor. The sensor converts this chemical signal into a recognizable electrical signal, thereby generating odor data samples.

[0050] Figure 4 A schematic diagram illustrating the determination of training loss according to an embodiment of the present disclosure is shown. Image sample x1 is input to image encoder 402 to obtain a first feature e1. Non-image sample x2 is input to non-image encoder 404 to obtain a second feature e2. Image encoder 402 propagates the first feature e1 to image decoder 406 and freshness prediction model 408 respectively, and non-image encoder 404 propagates the second feature e2 to freshness prediction model 408. Freshness prediction model 408 includes a cross-attention module and a prediction module. Image decoder 406 decodes the reconstructed image x' based on the first feature e1, and freshness prediction model 408 predicts the predicted label y' based on the first feature e1 and the second feature e2. A first reconstruction loss L1 can be obtained based on x1 and x', and a prediction loss L2 can be obtained based on the sample labels y and y'. L1 and L2 are weighted and summed according to preset weights to obtain training loss L3. The parameters of image encoder 402, image decoder 406, and freshness prediction model 408 are adjusted according to the training loss L3. In this embodiment, the image encoder, image decoder, cross-attention module, and prediction module can all be trained without the need for a pre-trained model.

[0051] Figure 5A schematic diagram of the structure of an image encoder according to an embodiment of the present disclosure is shown. In this embodiment, an image encoder consisting of network layers 502, 504, and 506 can be used to determine a first feature. This embodiment includes inputting a first representation into a convolutional layer 502 of the image encoder to obtain a first convolutional feature. This embodiment also includes inputting the first convolutional feature into a pooling layer 504 of the image encoder to obtain a first pooled feature. An exemplary pooling process is shown below pooling layer 504, which helps to achieve the purpose of adjusting dimensions. In this example, average pooling is used. This embodiment also includes inputting the first pooled feature into a fully connected layer 506 of the image encoder to obtain the first feature. Through the image encoder in this embodiment, image modality data can be effectively processed, features related to image information can be extracted, and dimensionality adjustment can be achieved. It should be noted that each network layer in this image encoder does not necessarily include only one layer, but may include multiple layers; for example, fully connected layer 506 shows two fully connected layers.

[0052] The image decoder can perform the corresponding inverse operation. In some embodiments, a first feature is input into an upsampling layer of the image decoder to obtain a first upsampled feature, and the first upsampled feature is input into a deconvolution layer of the image decoder to obtain a reconstructed image.

[0053] Figure 6 A schematic diagram of an application scenario according to an embodiment of the present disclosure is shown. In this embodiment, a refrigerator 602 represents a refrigeration device. This embodiment includes using a camera (not shown) to capture images of food 604 inside the refrigerator 602 as image samples. This embodiment also includes using a gas sensor or odor sensor (not shown) to capture gas data or odor data of a region 606 containing food 604 (i.e., the upper refrigerated compartment of the refrigerator) within the refrigerator 602 as non-image samples. In conjunction with other embodiments, the image and non-image samples thus acquired are used to train a model to predict the freshness of food 604. In some embodiments, a light sensor may also be used to capture light data about food 604 inside the refrigerator 602 as non-image samples. This application scenario is merely exemplary and does not constitute a limitation of the present disclosure.

[0054] Figure 7 A schematic diagram of a method for training a model according to an embodiment of the present disclosure is shown. This embodiment mainly relates to the processing of the cross-attention module. Figure 7 As shown, after the first feature e1 is encoded by the image encoder and the second feature e2 is encoded by the non-image encoder, the first feature e1 and the second feature e2 are propagated to the cross-attention module 700 (e.g., ...). Figure 4(As shown in 402-408). In the cross-attention module 700, a query matrix is ​​obtained at 702, where multiple query vectors are used to determine the focus of attention. A key matrix is ​​obtained at 704, which can be seen as clues provided to the query matrix for matching. A value matrix is ​​obtained at 706. The attention weights of each feature in the first feature e1 can be determined by the query matrix and the key matrix. The value matrix is ​​used to perform a weighted sum based on these weights to determine the corrected feature representation of the first feature e1. These matrices can be randomly initialized beforehand and continuously refined during the training process. After obtaining these matrices, a matrix operation is performed on the query matrix and the first feature e1 at 708 to obtain the first optimized feature. A matrix operation is performed on the key matrix and the first feature e1 at 710 to obtain the second optimized feature. At 714, an operation (e.g., a dot product) is performed on the first and second optimized features to obtain the weight matrix. This weight matrix indicates the weight of each element in the first feature e1. At 716, the weight matrix is ​​normalized. At position 712, matrix operations are performed on the value matrix and the second feature e2 to obtain the third optimized feature. At position 720, based on the third optimized feature and the normalized weight matrix, the fusion feature is determined.

[0055] In this embodiment, the fused feature can be the result of a weighted summation of the third optimized feature based on the weight matrix. It reflects those features in the first feature e1 and the second feature e2 that are related to the prediction task, that is, it achieves complementarity with the prediction-related features, while enhancing the features included in both the first feature e1 and the second feature e2. The fused feature has high accuracy and helps to improve the accuracy of prediction.

[0056] Figure 8A schematic diagram of a method for training a model according to another embodiment of the present disclosure is shown. In this embodiment, an image encoder 808 and a non-image encoder 810 can be trained simultaneously. In this embodiment, an image representation 804 is input into the image encoder 808 to obtain a first feature e1. A non-image representation 806 is input into the non-image encoder 810 to obtain a second feature e2. A first decoder 802 generates a reconstructed image 812 based on the first feature e1. A first reconstruction loss L1 is determined based on the reconstructed image 812 and image samples. A cross-attention module 814 and a prediction module 816 obtain predicted labels based on the first feature e1 and the second feature e2. This embodiment also includes generating reconstructed data 822 using a second decoder 820 based on the second feature e2. This embodiment also includes determining a prediction loss L2 based on the predicted label and sample label, determining a first reconstruction loss L1 based on the reconstructed image 812 and image samples, determining a second reconstruction loss L3 based on the reconstructed data 822 and non-image samples, determining a second training loss 818 based on the first reconstruction loss L1, the second reconstruction loss L3, and the prediction loss L2 (e.g., determined by weighted summation), and adjusting the parameters of the first encoder 808, the second encoder 810, the first decoder 802, the second decoder 820, the cross-attention module 814, and the prediction module 816 based on the second training loss 818. In other words, when calculating the total loss, the loss from the reconstructed data is taken into account, and then the parameters of the image decoder and the non-image decoder are adjusted together. According to the method of this embodiment, the ability of both encoders to retain valuable information and the encoding ability relevant to the prediction task can be trained simultaneously, which is beneficial for improving prediction accuracy.

[0057] Figure 9A This diagram illustrates a process flow for a data processing method according to an embodiment of the present disclosure, which corresponds to a model inference process. In this embodiment, an image encoder, a non-image encoder, and a task processing model trained using the methods described in the above embodiments can be employed. In this embodiment, image data and non-image data are acquired; the image data is converted into a first representation m1, and the non-image data is converted into a second representation m2. In this embodiment, the first representation m1 is encoded by the image encoder 902 to obtain a first feature n1, and the second representation m2 is encoded by the non-image encoder 904 to obtain a second feature n2. As analyzed above, the dimensions of the first feature n1 and the second feature n2 are aligned; for example, the dimension of n1 is 10×10, and the dimension of n2 is 10×6. Furthermore, the first feature n1 describes the image information in the first representation m1 related to the prediction task.

[0058] Thus, the first feature n1 and the second feature n2 are input into the cross-attention module 908 in the task processing model 906 to obtain the fused feature. This fused feature is then input into the prediction module 910 in the task processing model 906 to obtain the target predicted value 912. The method disclosed in this embodiment can achieve highly accurate predictions when completing the prediction task. Because the trained encoder already possesses the ability to extract core features (features of image information related to task processing) and alignment dimensions, the model's inference process only needs to complete the inference based on the first representation m1 and the second representation m2, thus eliminating the need to introduce a decoder to reconstruct the image or data.

[0059] Figure 9B A flowchart of a data processing method 900 according to another embodiment of the present disclosure is shown, which corresponds to the model inference process. In block 920, first data and second data are acquired, wherein the first data belongs to an image modality and the second data belongs to a non-image modality. During the inference phase, the modality of the data should be consistent with the modality of the samples during the training phase, which results in better prediction performance. In block 922, based on the first and second data, a target prediction value is determined using the trained first encoder, second encoder, and trained task processing model of the present disclosure.

[0060] In a scenario involving the detection of food freshness, the target predicted value indicates a prediction of the food's freshness. Images of the food within a refrigeration unit can be captured using a camera as first data, and gas or odor data of the area containing the food within the refrigeration unit can be collected using a gas sensor or odor sensor as second data. In this case, the method for processing the data can be a method for predicting the freshness of the food. As another example, the method of embodiments of this disclosure can be applied to medical devices. For example, image samples can be scanned images of target tissues of an organism, and non-image samples can be data indicating the change in potassium ion content of the target tissues of an organism over time. Predictive labels can indicate the health status of the target tissues of an organism. In this scenario, the first encoder and task processing model trained still have good predictive capabilities. In this case, the method for processing the data can be a method for predicting the health status of the target tissues of an organism.

[0061] Figure 10 This is a schematic diagram of an apparatus 1000 for training a model according to an embodiment of the present disclosure. Figure 10The illustrated apparatus 1000 includes a first prediction unit 1002 configured to determine a predicted label based on a first feature and a second feature using a task processing model. The first feature is determined by a first encoder based on a first sample belonging to an image modality, and the second feature is determined by a second encoder based on a second sample belonging to a non-image modality. The apparatus 1000 also includes a reconstruction unit 1004 configured to generate a reconstructed image based on the first feature using a first decoder. The apparatus 1000 further includes a training unit 1006 configured to train the first encoder, the first decoder, and the task processing model based on the predicted label, the reconstructed image, the first sample, and the sample label.

[0062] In some embodiments, the training unit 1006 includes a first loss determination unit configured to determine a prediction loss based on a predicted label and a sample label, wherein the sample label is the ground truth value corresponding to the first sample and the second sample. The training unit 1006 further includes a second loss determination unit configured to determine a first reconstruction loss based on the reconstructed image and the first sample. The training unit 1006 further includes a third loss determination unit configured to determine a first training loss based on the first reconstruction loss and the prediction loss. The training unit 1006 further includes a first adjustment unit configured to adjust the parameters of the first encoder, the first decoder, and the task processing model based on the first training loss.

[0063] In some embodiments, the third loss determination unit includes a first weight determination unit configured to obtain a first weight for the first reconstruction loss. The third loss determination unit further includes a second weight determination unit configured to obtain a second weight for the prediction loss. The third loss determination unit also includes a third loss determination subunit configured to determine a weighted sum of the first reconstruction loss and the prediction loss as a first training loss based on the first weight and the second weight.

[0064] In some embodiments, the apparatus 1000 further includes at least one of the following: a first feature is determined by a first encoder based on a first representation, the first representation indicating a first sample belonging to an image modality; a second feature is determined by a second encoder based on a second representation, the second representation indicating a second sample belonging to a non-image modality; the first representation and the second representation have different dimensions; and the first feature and the second feature have the same dimension; and the task processing model includes a cross-attention module.

[0065] In some embodiments, the reconstruction unit includes an upsampling unit configured to input a first feature into an upsampling layer of a first decoder to obtain a first upsampled feature. The reconstruction unit also includes a deconvolution unit configured to input the first upsampled feature into a deconvolution layer of the first decoder to obtain a reconstructed image.

[0066] In some embodiments, the apparatus 1000 further includes a second reconstruction unit configured to generate reconstructed data using a second decoder based on second features. The training unit 1004 includes a fourth loss determination unit configured to determine a prediction loss based on predicted labels and sample labels, where the sample labels are ground truth values ​​corresponding to the first and second samples. The training unit 1004 includes a fifth loss determination unit configured to determine a first reconstruction loss based on the reconstructed image and the first sample. The training unit 1004 includes a sixth loss determination unit configured to determine a second reconstruction loss based on the reconstructed data and the second sample. The training unit 1004 includes a seventh loss determination unit configured to determine a second training loss based on the first reconstruction loss, the second reconstruction loss, and the prediction loss. The training unit 1004 includes a second adjustment unit configured to adjust the parameters of the first encoder, the second encoder, the first decoder, the second decoder, and the task processing model based on the second training loss.

[0067] In some embodiments, the device 1000 further includes a first acquisition unit configured to acquire images of food within the refrigeration unit using a camera, as a first sample. The device 1000 also includes a second acquisition unit configured to acquire gas data or odor data of the area containing food within the refrigeration unit using a gas sensor or an odor sensor, as a second sample.

[0068] In some embodiments, where the prediction label indicates the freshness of the food, the device 1000 further includes a third acquisition unit configured to acquire regional light data of the food contained within the refrigeration device using a light sensor as a second sample.

[0069] Figure 11 This is a schematic diagram of an apparatus 1100 for processing data according to an embodiment of the present disclosure. Figure 11 The apparatus 1100 shown includes an acquisition unit 1102 configured to acquire first data and second data, wherein the first data belongs to an image modality and the second data belongs to a non-image modality. The apparatus 1100 also includes a second prediction unit 1104 configured to determine a target prediction value based on the first data and the second data, using a trained first encoder, a second encoder, and a trained task processing model according to this disclosure.

[0070] Figure 12A schematic block diagram of a controller 1200 suitable for implementing embodiments of the present disclosure is shown. As shown, the controller 1200 includes a processor 1201, which performs various appropriate actions and processes according to computer program instructions loaded into random access memory (RAM) 1203 based on computer program instructions stored in read-only memory (ROM) 1202. Various programs and data required for the operation of the controller 1200 may also be stored in RAM 1203. The processor 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. An input / output (I / O) interface 1205 is also connected to bus 1204.

[0071] The various methods and processes described above can be executed by processor 1201. For example, in some embodiments, the various methods and processes described above can be implemented as computer software programs tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program can be loaded into and / or installed onto controller 1200 via ROM 1202. When the computer program is loaded into RAM 1203 and executed by processor 1201, one or more actions of the methods and processes described above can be performed.

[0072] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0073] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), and any suitable combination thereof. The computer-readable storage medium as used herein is not to be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0074] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0075] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0076] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0077] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0078] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0080] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method (200) for training a model, comprising: Based on the first feature and the second feature, a predicted label is determined (202) by a task processing model, wherein the first feature is determined by a first encoder based on a first sample belonging to the image modality, and the second feature is determined by a second encoder based on a second sample belonging to the non-image modality; Based on the first feature, the first decoder generates (204) a reconstructed image; as well as Based on the predicted label, the reconstructed image, the first sample, and the sample label, train (206) the first encoder, the first decoder, and the task processing model.

2. The method according to claim 1, wherein training (206) the first encoder, the first decoder, and the task processing model based on the predicted label, the reconstructed image, the first sample, and the sample label comprises: The prediction loss is determined based on the predicted label and the sample label, wherein the sample label is the true value corresponding to the first sample and the second sample; A first reconstruction loss is determined based on the reconstructed image and the first sample; The first training loss is determined based on the first reconstruction loss and the prediction loss; as well as Based on the first training loss, adjust the parameters of the first encoder, the first decoder, and the task processing model.

3. The method of claim 2, wherein determining the first training loss based on the first reconstruction loss and the prediction loss comprises: Obtain the first weight for the first reconstruction loss; Obtain a second weight for the predicted loss; as well as Based on the first weight and the second weight, the weighted sum of the first reconstruction loss and the prediction loss is determined as the first training loss.

4. The method according to claim 1, further comprising at least one of the following: the first feature is determined by a first encoder based on a first representation, the first representation being used to indicate a first sample belonging to an image modality; the second feature is determined by a second encoder based on a second representation, the second representation being used to indicate a second sample belonging to a non-image modality; the first representation and the second representation have different dimensions; and the first feature and the second feature have the same dimension. The task processing model includes a cross-attention module.

5. The method of claim 1, wherein generating (204) a reconstructed image by the first decoder based on the first feature comprises: The first feature is input into the upsampling layer of the first decoder to obtain the first upsampled feature; as well as The first upsampled feature is input into the deconvolution layer of the first decoder to obtain the reconstructed image.

6. The method according to claim 1, further comprising: Reconstructed data is generated using a second decoder based on the second feature; And based on the predicted label, the reconstructed image, the first sample, and the sample label, training (206) the first encoder, the first decoder, and the task processing model includes: The prediction loss is determined based on the predicted label and the sample label, wherein the sample label is the true value corresponding to the first sample and the second sample; A first reconstruction loss is determined based on the reconstructed image and the first sample; The second reconstruction loss is determined based on the reconstructed data and the second sample; A second training loss is determined based on the first reconstruction loss, the second reconstruction loss, and the prediction loss; and The parameters of the first encoder, the second encoder, the first decoder, the second decoder, and the task processing model are adjusted based on the second training loss.

7. The method according to claim 1, further comprising: The first sample is obtained by capturing images of food inside a refrigeration device using a camera. Gas or odor data of the area containing the food within the refrigeration equipment is collected using a gas sensor or odor sensor as the second sample.

8. The method of claim 7, wherein the predictive label indicates the freshness of the food.

9. A method (900) for data processing, comprising: Acquire (920) first data and second data, wherein the first data belongs to the image modality and the second data belongs to the non-image modality; as well as Based on the first data and the second data, the target prediction value is determined using the first encoder, the second encoder, and the trained task processing model according to any one of claims 1-8.

10. The method of claim 9, wherein the target predicted value indicates a prediction of the freshness of the food, and acquiring the first data and the second data comprises: The food inside the refrigeration equipment is captured using a camera and used as the first data. The second data is obtained by using a gas sensor or odor sensor to collect gas data or odor data of the area containing the food within the refrigeration equipment, or by using a light sensor to collect light data of the area containing the food within the refrigeration equipment.

11. An apparatus (1000) for training a model, comprising: The first prediction unit (1002) is configured to determine a predicted label based on a first feature and a second feature through a task processing model, wherein the first feature is determined by a first encoder based on a first sample belonging to an image modality, and the second feature is determined by a second encoder based on a second sample belonging to a non-image modality. The reconstruction unit (1004) is configured to generate a reconstructed image by a first decoder based on the first feature; as well as The training unit (1006) is configured to train the first encoder, the first decoder, and the task processing model based on the predicted label, the reconstructed image, the first sample, and the sample label.

12. An apparatus (1100) for processing data, comprising: The acquisition unit (1102) is configured to acquire first data and second data, wherein the first data belongs to an image modality and the second data belongs to a non-image modality; as well as The second prediction unit (1104) is configured to determine a target prediction value based on the first data and the second data, using the first encoder, the second encoder, and the trained task processing model as described in any one of claims 1-8.

13. A controller, comprising: At least one processor; as well as A memory coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, cause the controller to perform the method according to any one of claims 1-12.

14. A refrigeration device comprising the controller according to claim 13.

15. A computer program product tangibly stored on a non-transient computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 10.

Images