Sample image acquisition method, model updating method and related equipment
By calculating the difference between the image reconstruction model on the image in the dishwasher, determining the sample image and retraining the model, the problem of poor detection effect of the deep learning model under the new data distribution is solved, and the performance and adaptability of the model are improved.
Patent Information
- Application Number
- CN202310604277.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-23
- Publication Date
- 2024-11-26
AI Technical Summary
Deep learning models are difficult to obtain good detection results when processing images of new data distribution, especially in scenarios such as dishwashers, because it is difficult to obtain training data covering all tableware combinations and placement habits in advance.
By acquiring the original image and using the image reconstruction model for encoding and decoding, the difference between the original image and the restored image is calculated, and it is determined as a sample image, and using these sample images to retrain the image reconstruction model and image detection model to improve the detection of the model and rebuild performance.
Effectively mine the data distribution different from previous sample images, improve the detection accuracy of the image detection model and the reconstruction accuracy of the image reconstruction model, and enhance the model's adaptability to new data distribution.
Smart Images

Figure CN119027753A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent product technology development, and in particular to a sample image acquisition method, a model updating method and related equipment. Background Art
[0002] With the development of intelligent technology, deep learning models have been used in all aspects of people's production and life. Among them, deep learning models rely on a large amount of training data. However, in some scenarios, it is difficult to obtain training data that can cover various situations in the scenario in advance.
[0003] For example, for a dishwasher, an image detection model can be used to detect the tableware in the dishwasher, so that the detection results can be used to automatically control the dishwasher to wash. However, since different families use different tableware and have different tableware placement habits, it is difficult to obtain training images that can cover all tableware combinations and placement habits in the dishwasher in advance. Therefore, when processing images that follow the new data distribution, the image detection model often finds it difficult to obtain detection results that meet the requirements.
[0004] Therefore, how to automatically mine sample images that obey the new data distribution and improve model performance is a problem that needs to be solved urgently. Summary of the invention
[0005] The main technical problem solved by the present application is to provide a sample image acquisition method, a model updating method and related equipment, which can automatically mine sample images that obey the new data distribution, thereby improving the performance of the image reconstruction model and the image detection model.
[0006] To solve the above technical problems, a technical solution adopted in the present application is: to provide a sample image acquisition method, the method comprising: obtaining an original image of a target object; using an image reconstruction model to perform encoding and decoding processing on the original image in sequence to obtain a restored image of the original image; obtaining a first difference between the original image and the restored image; based on the first difference, determining whether to use the original image as a first sample image; wherein the image reconstruction model and the image detection model are pre-trained based on the same second sample image, and the first sample image is subsequently used to retrain the image reconstruction model and the image detection model.
[0007] The first difference includes the accumulated value of the absolute difference between the corresponding pixels of the original image and the restored image. Based on the first difference, determining whether to use the original image as the first sample image includes: in response to the accumulated value reaching a first preset threshold, determining to use the original image as the first sample image.
[0008] The sample image acquisition method further includes: in response to the number of first sample images reaching a second preset threshold, retraining the image reconstruction model and the image detection model using each first sample image.
[0009] Among them, before using the image reconstruction model to encode and decode the original image in sequence to obtain the restored image of the original image, the sample image acquisition method also includes the following pre-training step of the image reconstruction model: using the image reconstruction model to encode and decode the second sample image in sequence to obtain a sample restored image of the second sample image; using the second difference between the sample restored image and the second sample image, adjusting the network parameters of the image reconstruction model.
[0010] Among them, the image reconstruction model includes an encoder and a decoder; the encoder includes a plurality of sequentially connected downsampling units, and the decoder includes a plurality of sequentially connected upsampling units and convolution units; the image reconstruction model is used to sequentially encode and decode the original image to obtain a restored image of the original image, including: the original image is sequentially passed through a plurality of sequentially connected downsampling units to obtain a first feature map output by each downsampling unit; the first feature map output by the last downsampling unit is sequentially passed through a plurality of sequentially connected upsampling units to obtain a second feature map output by each upsampling unit; the convolution unit is used to convolve the second feature map output by the last upsampling unit to obtain a restored image; wherein the upsampling unit and the downsampling unit are pre-correlated one-to-one, and for the next upsampling unit of any upsampling unit, the input of the next upsampling unit is a fusion feature of the second feature map output by the upsampling layer and the first feature map of the corresponding downsampling unit.
[0011] The target object is the items to be cleaned placed in the dishwasher. The dishwasher is provided with an image acquisition element for acquiring the original image. The image detection model is used to detect the original image to obtain the cleaning strategy for the items to be cleaned.
[0012] Wherein, acquiring the original image of the target object includes: in response to receiving a door closing signal of the dishwasher, initializing the image acquisition element so that the image acquisition element acquires the original image of the target object; and receiving the original image sent by the image acquisition element.
[0013] To solve the above technical problems, another technical solution adopted in the present application is: to provide an image detection model updating method, the method comprising: obtaining a first sample image; wherein the first sample image is acquired by any one of the sample image acquisition methods; and using the first sample image to update the image detection model to obtain a new image detection model.
[0014] The first sample image is obtained by photographing items to be cleaned in the dishwasher and is marked with a sample cleaning strategy that matches the items to be cleaned in the first sample image. After using the first sample image to update the image detection model to obtain a new image detection model, the method also includes: sending the new image detection model to the dishwasher; the dishwasher detects the newly taken original image based on the new image detection model to obtain a matching cleaning strategy.
[0015] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising a memory and a processor coupled to each other, the memory storing program instructions; the processor is used to execute the program instructions stored in the memory to implement any one of the sample image acquisition methods or the image detection model update methods.
[0016] In order to solve the above-mentioned technical problems, another technical solution adopted in the present application is: to provide a dishwasher, including a dishwasher body, an image acquisition element and a main control module, the image acquisition element and the main control module are arranged on the dishwasher body, the image acquisition element is electrically connected to the main control module, the image acquisition element is used to photograph the items to be cleaned in the dishwasher to obtain the original image, the main control module is used to execute any one of the sample image acquisition methods, and run the image detection model to detect the original image to obtain a cleaning strategy that matches the items to be cleaned.
[0017] To solve the above technical problems, another technical solution adopted in the present application is: to provide a server, including a memory and a processor coupled to each other, the memory storing program instructions, and the processor being used to execute the program instructions stored in the memory to implement any method of the image detection model updating method.
[0018] In order to solve the above technical problems, another technical solution adopted in the present application is: to provide a model updating system, including the above-mentioned dishwasher and server, the dishwasher and the server are communicatively connected, and the server is used for any one of the image detection model updating methods.
[0019] To solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium, which is used to store program instructions, and the program instructions can be executed to implement any one of the sample image acquisition method and the image detection model update method.
[0020] The above method, when the image reconstruction model and the image detection model are pre-trained based on the same second sample image, uses the first difference between the original image and the restored image reconstructed by the image reconstruction model to determine whether to use the original image as the first sample image for subsequent retraining of the above two models. Since the first difference between the original image before and after reconstruction is evaluated by the image reconstruction model, the original image with a data distribution different from the previous one can be mined and used as the first sample image, so that the image detection model can be retrained by the first sample image, and the detection accuracy of the image detection model can be improved as much as possible. At the same time, since the image reconstruction model can be retrained by the first sample image, the reconstruction accuracy of the image reconstruction model can be improved as much as possible, which helps to further mine the original image with a data distribution different from the previous one through the image reconstruction model, and continuously mine the original image with a data distribution different from the previous one as much as possible, improve the richness of the sample image, and then continuously promote the reconstruction accuracy of the image reconstruction model and the detection accuracy of the image detection model. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is a flow chart of an embodiment of a sample image acquisition method provided by the present application;
[0022] Figure 2 It is a schematic diagram of the process of obtaining a restored image of an original image by the image reconstruction model provided by the present application;
[0023] Figure 3 It is a flowchart of an embodiment of an image detection model updating method provided by the present application;
[0024] Figure 4 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;
[0025] Figure 5 is a schematic diagram of a frame of an embodiment of a dishwasher provided by the present application;
[0026] Figure 6 It is a schematic diagram of the framework of an embodiment of a server provided by the present application;
[0027] Figure 7 It is a schematic diagram of the framework of an embodiment of a model updating system provided by the present application;
[0028] Figure 8 It is a schematic diagram of the framework of the computer-readable storage medium provided by this application. DETAILED DESCRIPTION
[0029] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.
[0030] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0031] See also Figure 1 , Figure 1 is a flow chart of an embodiment of a sample image acquisition method provided by the present application. It should be noted that if there are substantially the same results, this embodiment is not used. Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:
[0032] S11: Acquire the original image of the target object.
[0033] This embodiment is used to obtain a first difference of a corresponding restored image by processing an original image and an image reconstruction model, and collect a first sample image for subsequent retraining of an image reconstruction model and an image detection model by using the first sample image.
[0034] It should be noted that the execution subject of this embodiment is an electronic device or a server that is in communication with the electronic device, which can be determined according to the actual scenario and is not specifically limited here. In one embodiment, the execution subject is an electronic device, and the electronic device at least includes an image reconstruction model; of course, in other embodiments, the execution subject is a server that is in communication with the electronic device, and the server at least includes an image reconstruction model.
[0035] It should also be noted that the target object described in this article is an item to be processed placed in an electronic device. In some scenarios, different users have different target objects, and the combinations of target objects placed in electronic devices are also different. It is difficult to obtain in advance relevant training data that can cover all target objects and combinations of target objects in the electronic device. Therefore, a model trained using only a small amount of pre-obtained training data generally cannot obtain a processing effect that meets the requirements when processing new data. If the processing effect of the model is to be improved, a sufficient amount of training data needs to be provided. The sample image acquisition method provided in this embodiment can be used to collect new training data for a model that lacks training data to increase the training data of the model, thereby facilitating the subsequent retraining of the model using the newly added training data to obtain a new model.
[0036] The electronic device described above may be, but is not limited to, a dishwasher or a refrigerator. In one embodiment, the electronic device is a dishwasher, and the target object is an object to be washed placed in the dishwasher, such as a plate, a dish, a knife or a fruit. The dishwasher is provided with an image acquisition element for collecting the original image of the target object. The image acquisition element may be, but is not limited to, a camera device, or an infrared device, and the number of image acquisition elements may be set according to actual needs. In one embodiment, a trigger is provided in the dishwasher. When a door closing signal of the dishwasher is received, the image acquisition element may be initialized so that the image acquisition element collects the original image of the target object, and then subsequent processing is performed according to the original image of the object to be washed collected by the image acquisition element. Of course, in other embodiments, a touch key for controlling the initialization of the image acquisition element is provided on the outer surface of the dishwasher. After the user triggers the touch key, the dishwasher will receive the touch signal and initialize the image acquisition element to collect the original image of the target object. The number of image acquisition elements is at least one, and the original image obtained may be obtained by fusion of images collected by multiple image acquisition elements. The specific number of image acquisition elements may be determined according to the effect of the original image actually obtained, and is not specifically limited here.
[0037] Of course, in other embodiments, the electronic device may also be other devices such as a refrigerator, and the target object may be various foods placed in the refrigerator, etc. The specific electronic device and the target object placed in the specific electronic device may be determined according to actual needs and are not specifically limited here.
[0038] S12: using the image reconstruction model to perform encoding and decoding processing on the original image in sequence to obtain a restored image of the original image.
[0039] In this embodiment, the image reconstruction model is used to perform encoding and decoding processing on the original image after acquiring the original image to obtain a restored image of the original image.
[0040] In one embodiment, the image reconstruction model includes an encoder and a decoder, wherein the encoder is used to perform multiple downsampling of the original image to extract the feature map (high-dimensional semantic features) of the original image, for example, converting the original image of 640x480x3 into a feature map of 20x15x1024; the decoder is used to upsample the extracted feature map to restore it to a restored image corresponding to the original image. Exemplarily, the decoder continuously convolves and upsamples the extracted high-dimensional semantic features to obtain a restored image of the original image. In one embodiment, in order to maintain better feature map details and avoid edge jaggedness or stacking effects caused by deconvolution operations, the feature map can be enlarged by upsampling and convolution of bilinear difference.
[0041] In a specific embodiment, in order to enhance more semantic information of the original image, an image reconstruction model with a U-Net structure can be used to splice the features output by the encoder module to the features output by the corresponding decoder module, so that the image reconstruction model can collect more image features and the restored image can be better.
[0042] The image reconstruction model of the U-Net structure is an encoder-decoder structure model, where the encoder includes a plurality of sequentially connected downsampling units, and the decoder includes a plurality of sequentially connected upsampling units and convolution units. The upsampling units correspond to the downsampling units in advance. Figure 2 , Figure 2 FIG. 1 is a flow chart of the image reconstruction model provided by the present application to obtain a restored image of the original image. Figure 2 As shown, in this embodiment, the original image is first passed through a plurality of sequentially connected downsampling units in sequence to obtain a first feature map output by each downsampling unit; then the first feature map output by the last downsampling unit is passed through a plurality of sequentially connected upsampling units in sequence to obtain a second feature map output by each upsampling unit; after obtaining each first feature map and the second feature map, a convolution unit is used to perform convolution processing on the second feature map output by the last upsampling unit to obtain a restored image of the original image, wherein, for the next upsampling unit of any upsampling unit, the input of the next upsampling unit is a fusion feature of the second feature map output by the upsampling layer and the first feature map of the corresponding downsampling unit.
[0043] S13: Obtain a first difference between the original image and the restored image.
[0044] In this embodiment, after obtaining the restored image of the original image, a first difference between the original image and the restored image is obtained to determine the restoration effect of the image reconstruction model according to the first difference. The larger the first difference, the worse the restoration effect of the original image, and the smaller the first difference, the better the restoration effect of the original image.
[0045] It should be noted that the original image and the restored image have the same size, and the pixels of the original image and the pixels of the restored image correspond one to one. In one embodiment, the first difference can be represented by the accumulated value of the absolute difference between the corresponding pixels of the original image and the restored image. Exemplarily, the accumulated value of the absolute difference between the corresponding pixels of the original image and the restored image can be obtained by the following formula:
[0046]
[0047] Where loss represents the first difference, c represents the value of each color channel in the three color channels of the pixel point, ij represents the pixel point located in the i-th row and j-th column, x represents the pixel value of the restored image, and y represents the pixel value of the original image.
[0048] S14: Based on the first difference, determine whether to use the original image as the first sample image; wherein the image reconstruction model and the image detection model are pre-trained based on the same second sample image, and the first sample image is subsequently used to retrain the image reconstruction model and the image detection model.
[0049] In one embodiment, when the first difference (the accumulated value of the absolute difference between the corresponding pixels of the original image and the restored image) reaches (is greater than or equal to) the first preset threshold, the original image is determined as the first sample image. The first preset threshold can be set according to actual needs and is not specifically limited here. For example, if more first sample images need to be obtained, the first preset threshold can be set to a larger value; if a better processing effect of the model is required, the first preset threshold can also be set to a smaller value.
[0050] It should be noted that before step S12, or even before step S11, the image reconstruction model and the image detection model need to be pre-trained using the same second sample image to obtain the pre-trained image reconstruction model and the image detection model.
[0051] The following takes the pre-training steps of the image reconstruction model as an example to illustrate the training process of the image reconstruction model and the image detection model. Specifically, the pre-training steps of the image reconstruction model are as follows:
[0052] First, the second sample image is encoded and decoded in sequence using the image reconstruction model to obtain a sample restored image of the second sample image.
[0053] Second, using a second difference between the sample restored image and the second sample image, the network parameters of the image reconstruction model are adjusted.
[0054] Exemplarily, the model can be guided to be trained in back propagation according to the second difference until the model converges.
[0055] Among them, the training process of the image reconstruction model and the image detection model both use the same second sample image for training. The difference is that the image detection model first detects the second sample image to obtain the corresponding sample detection result, and then uses the difference between the labeled detection result and the sample detection result to adjust the network parameters of the image detection model until the model converges.
[0056] From the above process, it can be seen that the pre-trained image reconstruction model and image detection model can accurately process the second sample image and obtain a processing result that meets the requirements. Exemplarily, for any second sample image, the second difference between the sample restored image restored by the pre-trained image reconstruction model and the second sample image meets the preset requirement (less than a preset threshold).
[0057] Therefore, when the image reconstruction model processes the original image and the first difference between the restored image and the original image reaches (is greater than or equal to) the first preset threshold, it means that the data distribution of the original picture currently processed by the image reconstruction model and the second sample image during the previous pre-training are images with different data distributions. In this case, the image detection model detects the original image and the detection result obtained is also inaccurate. Therefore, the original image with a different data distribution from the previous second sample image can be collected as the first sample image, and the image reconstruction model and the image detection model can be retrained using the collected first sample image to update the image reconstruction model and the image detection model. It can be understood that the updated image reconstruction model and the image detection model can obtain better processing effects when processing images with the same or similar data distribution as the first sample image.
[0058] In one embodiment, when the number of collected first sample images reaches a second preset threshold, the image reconstruction model and the image detection model can be retrained using each first sample image. The specific second preset threshold can be determined based on actual conditions and actual needs and is not specifically limited here.
[0059] In this embodiment, the image detection model is another processing model different from the image reconstruction model. When the device is a dishwasher, the main function of the image detection model is to detect the original image and obtain the cleaning strategy for the items to be cleaned; of course, when the device is a refrigerator, the main function of the image detection model is to detect the original image and obtain the control strategy for the items to be kept fresh and / or frozen. The specific image detection model can be determined according to the specific device and device needs, and is not specifically limited here.
[0060] The above method, when the image reconstruction model and the image detection model are pre-trained based on the same second sample image, uses the first difference between the original image and the restored image reconstructed by the image reconstruction model to determine whether to use the original image as the first sample image for subsequent retraining of the above two models. Since the first difference between the original image before and after reconstruction is evaluated by the image reconstruction model, the original image with a data distribution different from the previous one can be mined and used as the first sample image, so that the image detection model can be retrained by the first sample image, and the detection accuracy of the image detection model can be improved as much as possible. At the same time, since the image reconstruction model can be retrained by the first sample image, the reconstruction accuracy of the image reconstruction model can be improved as much as possible, which helps to further mine the original image with a data distribution different from the previous one through the image reconstruction model, and continuously mine the original image with a data distribution different from the previous one as much as possible, improve the richness of the sample image, and then continuously promote the reconstruction accuracy of the image reconstruction model and the detection accuracy of the image detection model.
[0061] See also Figure 3 , Figure 3 is a flow chart of an embodiment of the image detection model updating method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not use Figure 3 The process sequence shown is limited. Figure 3 As shown, this embodiment includes:
[0062] S31: Acquire a first sample image.
[0063] This embodiment is used to update the image detection model using the first sample image after a new training sample image (first sample image) is collected to obtain a new image detection model.
[0064] The execution subject of this embodiment is the above-mentioned electronic device or a server that is in communication connection with the electronic device. The specific execution subject can be determined according to the actual scenario and is not specifically limited here.
[0065] The first sample image is acquired by any of the above sample image acquisition methods. The specific first sample image acquisition process can refer to the above steps S11 to S14, which will not be described in detail here.
[0066] S32: Use the first sample image to update the image detection model to obtain a new image detection model.
[0067] In this embodiment, after obtaining the first sample image, the first sample image will be annotated, and then the first sample image will be detected using the image detection model to obtain the detection result corresponding to the first sample image. Then, the difference between the annotation of the first sample image and the corresponding detection result is used to guide the network to train in back propagation to update the image detection model and obtain a new image detection model.
[0068] In a specific embodiment, the electronic device is a dishwasher, and the first sample image is obtained by photographing the items to be cleaned (target objects) in the dishwasher, and is annotated with a sample cleaning strategy that matches the items to be cleaned in the first sample image. Therefore, in this embodiment, the image detection model can be used to detect the first sample image to obtain the detection result (cleaning strategy) corresponding to the first sample image, and then the difference between the sample cleaning strategy matching the items to be cleaned in the first sample image and the detection result is used to update the image detection model to obtain a new image detection model.
[0069] In a specific embodiment, the execution subject corresponding to steps S31 and S32 is a server connected to the electronic device, and the electronic device is a dishwasher. When the server obtains the first sample image, and updates the image detection model using the first sample image to obtain a new image detection model, the new image detection model is sent to the dishwasher, and the dishwasher detects the newly captured original image based on the new image detection model to obtain a matching cleaning strategy, thereby facilitating the dishwasher to use the matching cleaning strategy to clean the items to be cleaned. Of course, in other embodiments, the server may not send the new image detection model to the dishwasher, but after obtaining the newly captured original image sent by the dishwasher, directly detect the newly captured original image based on the new image detection model to obtain a matching cleaning strategy, and send the matching cleaning strategy to the dishwasher, so that the dishwasher uses the matching cleaning strategy to clean the items to be cleaned. Among them, the cleaning strategy is, for example, at least one of the water output, water output intensity, detergent dosage, and water output direction, or a combination of at least one.
[0070] See also Figure 4 , Figure 4 4 is a schematic diagram of a framework of an electronic device according to an embodiment of the present application. In this embodiment, the electronic device 40 includes a memory 41 and a processor 42 coupled to each other.
[0071] The memory 41 stores program instructions, and the processor 42 is used to execute the program instructions stored in the memory 41 to implement the steps of any method in the above-mentioned sample image acquisition method, or to implement the steps of any method in the above-mentioned image detection model update method. In a specific implementation scenario, the electronic device 40 may include, but is not limited to: a dishwasher, a refrigerator, or a server. In addition, the electronic device 40 may also include mobile devices such as a laptop computer and a tablet computer, which are not limited here.
[0072] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps of any of the above-mentioned embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.
[0073] See also Figure 5 , Figure 5 : is a schematic diagram of a framework of an embodiment of a dishwasher provided by the present application. In this embodiment, the dishwasher 50 includes a dishwasher body 51, an image acquisition element 52 and a main control module (not shown in the figure), the image acquisition element 52 and the main control module are arranged on the dishwasher body 51, the image acquisition element 52 is electrically connected to the main control module, the image acquisition element 52 is used to photograph the items to be cleaned in the dishwasher 50 to obtain the original image, and the main control module is used to execute the method of any implementation method of the sample image acquisition method and run the image detection model to detect the original image and obtain a cleaning strategy that matches the items to be cleaned.
[0074] See also Figure 6 , Figure 6 6 is a schematic diagram of a framework of an embodiment of a server provided by the present application. In this embodiment, the server 60 includes a memory 61 and a processor 62 coupled to each other. The memory 61 stores program instructions, and the processor 62 is used to execute the program instructions stored in the memory 61 to implement any of the methods for updating the above-mentioned image detection model.
[0075] In a specific implementation scenario, the server 60 may be connected to a dishwasher or a refrigerator for communication. The server 60 may include a mobile device such as a laptop computer or a tablet computer, which is not limited here.
[0076] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps of any embodiment of the updating method of the image detection model. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.
[0077] See also Figure 7 , Figure 7 1 is a schematic diagram of a framework of an embodiment of a model updating system provided by the present application. In this embodiment, the model updating system 70 includes a dishwasher 71 (the above-mentioned dishwasher 50) and a server 72 (the above-mentioned server 60), and the dishwasher 71 and the server 72 are in communication connection.
[0078] See also Figure 8 , Figure 8 It is a schematic diagram of the framework of the computer-readable storage medium provided by the present application. The computer-readable storage medium 80 of the embodiment of the present application stores a program instruction 81, and when the program instruction 81 is executed, the method provided by any embodiment of the above method and any non-conflicting combination is implemented. Among them, the program instruction 81 can form a program file and be stored in the above-mentioned computer-readable storage medium 80 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 80 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.
[0079] The above method, when the image reconstruction model and the image detection model are pre-trained based on the same second sample image, uses the first difference between the original image and the restored image reconstructed by the image reconstruction model to determine whether to use the original image as the first sample image for subsequent retraining of the above two models. Since the first difference between the original image before and after reconstruction is evaluated by the image reconstruction model, the original image with a data distribution different from the previous one can be mined and used as the first sample image, so that the image detection model can be retrained by the first sample image, and the detection accuracy of the image detection model can be improved as much as possible. At the same time, since the image reconstruction model can be retrained by the first sample image, the reconstruction accuracy of the image reconstruction model can be improved as much as possible, which helps to further mine the original image with a data distribution different from the previous one through the image reconstruction model, and continuously mine the original image with a data distribution different from the previous one as much as possible, improve the richness of the sample image, and then continuously promote the reconstruction accuracy of the image reconstruction model and the detection accuracy of the image detection model.
[0080] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0081] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.
[0082] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0083] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0084] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0085] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
[0086] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A sample image acquisition method, characterized in that: The method comprises: Obtain the original image of the target object; Using an image reconstruction model to sequentially perform encoding and decoding processing on the original image to obtain a restored image of the original image; Obtaining a first difference between the original image and the restored image; Based on the first difference, determine whether to use the original image as a first sample image; wherein the image reconstruction model and the image detection model are pre-trained based on the same second sample image, and the first sample image is subsequently used to retrain the image reconstruction model and the image detection model.
2. The method according to claim 1, characterized in that The first difference includes an accumulated value of absolute differences between corresponding pixels of the original image and the restored image, and determining whether to use the original image as the first sample image based on the first difference includes: In response to the accumulated value reaching a first preset threshold, it is determined to use the original image as the first sample image.
3. The method according to claim 1 or 2, characterized in that: The method further comprises: In response to the number of the first sample images reaching a second preset threshold, the image reconstruction model and the image detection model are retrained using the first sample images.
4. The method according to claim 1, characterized in that: Before the original image is encoded and decoded in sequence by using the image reconstruction model to obtain a restored image of the original image, the method further includes the following pre-training step of the image reconstruction model: Using the image reconstruction model, sequentially encoding and decoding the second sample image to obtain a sample restored image of the second sample image; The network parameters of the image reconstruction model are adjusted using a second difference between the sample restored image and the second sample image.
5. The method according to claim 1, characterized in that The image reconstruction model includes an encoder and a decoder; the encoder includes a plurality of sequentially connected downsampling units, and the decoder includes a plurality of sequentially connected upsampling units and convolution units; The using the image reconstruction model to sequentially perform encoding and decoding processing on the original image to obtain a restored image of the original image includes: Passing the original image through the plurality of sequentially connected down-sampling units in sequence, to obtain a first feature map outputted by each down-sampling unit; Passing the first feature map output by the last down-sampling unit through the plurality of sequentially connected up-sampling units in sequence, to obtain second feature maps output by each up-sampling unit; Using the convolution unit to perform convolution processing on the second feature map output by the last upsampling unit to obtain the restored image; Among them, the upsampling units and the downsampling units are pre-correlated one-to-one, and for the next upsampling unit of any upsampling unit, the input of the next upsampling unit is the fusion feature of the second feature map output by the upsampling layer and the first feature map of the corresponding downsampling unit.
6. The method according to any one of claims 1 to 5, characterized in that: The target object is an item to be cleaned placed in a dishwasher. The dishwasher is provided with an image acquisition element for acquiring the original image. The image detection model is used to detect the original image to obtain a cleaning strategy for the item to be cleaned.
7. The method according to claim 6, characterized in that The obtaining of the original image of the target object comprises: In response to receiving a door closing signal of the dishwasher, initializing the image acquisition element so that the image acquisition element acquires the original image of the target object; Receive the original image sent by the image acquisition component.
8. A method for updating an image detection model, characterized in that: include: Acquire a first sample image; wherein the first sample image is acquired by the method according to any one of claims 1 to 7; The image detection model is updated using the first sample image to obtain a new image detection model.
9. The updating method according to claim 8, characterized in that: The first sample image is obtained by photographing the items to be cleaned in the dishwasher and is annotated with a sample cleaning strategy that matches the items to be cleaned in the first sample image. After the image detection model is updated using the first sample image to obtain a new image detection model, the method further includes: The new image detection model is sent to the dishwasher; wherein the dishwasher detects the newly captured original image based on the new image detection model to obtain a matching cleaning strategy.
10. An electronic device, characterized in that: comprising a memory and a processor coupled to each other, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method described in any one of claims 1-7 or 8-9.
11. A dishwasher, characterized in that: The dishwasher comprises a dishwasher body, an image acquisition element and a main control module, wherein the image acquisition element and the main control module are arranged on the dishwasher body, the image acquisition element is electrically connected to the main control module, the image acquisition element is used to photograph the objects to be cleaned in the dishwasher to obtain an original image, and the main control module is used to execute the method according to any one of claims 1 to 7 and run an image detection model to detect the original image to obtain a cleaning strategy matching the objects to be cleaned.
12. A server, characterized in that: A memory and a processor coupled to each other, the memory storing program instructions, and the processor being used to execute the program instructions stored in the memory to implement the method described in any one of claims 8-9.
13. A model updating system, characterized in that: The dishwasher according to claim 11 and the server according to claim 12 are included, and the dishwasher and the server are communicatively connected.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program instructions, and the program instructions can be executed to implement the method according to any one of claims 1 to 9.
