Cooking video processing method and device, cooking equipment and storage medium

Abstract

The present application relates to the technical field of image processing, and discloses a cooking video processing method and device, a cooking device and a storage medium, the method comprising: calculating a difference value of each frame of cooking image recorded by the cooking device in a current cooking video recording period and a previous cooking storage image respectively; judging whether the average of the difference values of each frame of cooking image recorded in the current cooking video recording period and the previous cooking storage image is greater than a preset difference value threshold; when the average of the difference values of each frame of cooking image recorded in the current cooking video recording period and the previous cooking storage image is greater than the preset difference value threshold, extracting a first cooking image from the cooking image recorded in the current cooking video recording period for storage; and splicing all the cooking images stored by the cooking device in the entire cooking process according to storage time to obtain a target cooking video. The present application reduces the requirement of the cooking device on storage under the condition of meeting the user's demand, and further reduces the production cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a cooking video processing method and device, a cooking equipment and a storage medium. BACKGROUND

[0002] With the popularity of cooking equipment such as steam ovens and the development of related technologies, its functions are becoming more and more rich, and the demand for intelligentization of users is becoming more and more strong. Some steam ovens are equipped with a camera module that can record a video of the entire cooking process. Usually, a cooking process takes at least ten minutes, and the working time of most intelligent menus is about 30 minutes. This results in a long recording time of the original video captured by the camera, and after recording the entire cooking process, the video file will be very large. If this function needs to be met, the camera module needs to be configured with a very large storage. In view of the cost, it is necessary to reduce the size of the saved video image, reduce the demand for storage, and reduce the design cost.

[0003] In related technologies, the following mainstream technologies are used to reduce the file size of the video: selecting a suitable encoder, adjusting the resolution and frame rate, code rate control, and compression format selection. The above-mentioned technologies can reduce the size of the recorded video to a certain extent, but on the basis of meeting the basic clarity, the above-mentioned technologies can only reduce the size of the video file to a certain extent, and the finally generated video is still large. If the compression is too much, the clarity of the video cannot be guaranteed, and the user's display needs cannot be met. SUMMARY

[0004] Therefore, the present application provides a cooking video processing method and device, a cooking equipment and a storage medium to solve the problem that the video file recorded in the cooking process occupies a large storage space in related technologies, and it is difficult to guarantee the clarity of the video and meet the user's display needs.

[0005] In a first aspect, the present application provides a cooking video processing method applied to a cooking equipment, the method comprising:

[0006] calculating a difference value of each frame of cooking image recorded in a current cooking video recording period and a previous cooking storage image of the cooking equipment respectively;

[0007] determining whether the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the previous cooking storage image is greater than a preset difference value threshold;

[0008] when the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the previous cooking storage image is greater than the preset difference value threshold, extracting a first cooking image from the cooking image recorded in the current cooking video recording period for storage;

[0009] Splice all cooking images stored by the cooking device during the entire cooking process according to storage time to obtain a target cooking video.

[0010] The present application compares the average of the difference values of the cooking images recorded in the current cooking video recording period and the last cooking stored image with the preset difference value threshold size, and selects cooking images from the current cooking video recording period for storage when the difference value is larger than the preset difference value threshold. This cooking image selection method not only meets the user's demand to show the change of food materials during cooking, but also greatly reduces the number of stored cooking images by selecting part of the cooking images, thereby reducing the file size of the target cooking video obtained by splicing, and thus reducing the storage requirements of the cooking device and the production cost of the cooking device while meeting the user's demand.

[0011] In an optional embodiment, when the average of the difference values of each frame of cooking image recorded in the current cooking video recording period and the last cooking stored image is not greater than the preset difference value threshold, the next frame of cooking image recorded outside the current cooking video recording period is added to the current cooking video recording period, the current cooking video recording period is updated, and the step of calculating the difference value of each frame of cooking image recorded in the current cooking video recording period and the last cooking stored image is returned.

[0012] When the average of the difference values of each frame of cooking image recorded in the current cooking video recording period and the last cooking stored image is small, the present application does not store the cooking image, but continuously updates the current cooking video recording period by adding newly recorded cooking images to the current cooking video recording period, and compares the average of the difference values again, so as to further ensure that the finally stored cooking images are cooking images with significant image changes, and further reduce the data amount of the cooking video while ensuring that the cooking video contains key images with significant changes of food materials during cooking, thereby improving the user experience.

[0013] In an optional embodiment, the method further comprises:

[0014] When the current cooking video recording period reaches the preset maximum cooking video recording period, a second cooking image is extracted from the cooking images recorded in the current cooking video recording period for storage.

[0015] The application selects and stores the cooking images collected in the process, avoids the cooking video from being oversimplified due to slow change of the food materials in the cooking process, and further ensures the completeness of the cooking video, meets the actual needs of the user for the cooking video, and improves the user experience.

[0016] In an optional embodiment, after the first cooking image or the second cooking image is extracted from the cooking images recorded in the current cooking video recording period and stored, the method further comprises:

[0017] deleting the cooking images recorded in the current cooking video recording period.

[0018] The application deletes the cooking video recorded in the current cooking video recording period after the cooking video is extracted and stored, reduces the data cache space required for recording the cooking video in the cooking process, further reduces the requirement for the data storage capacity of the cooking device, and reduces the cost.

[0019] In an optional embodiment, the difference value of each frame of the cooking image recorded in the current cooking video recording period of the cooking device and the previous cooking storage image comprises:

[0020] calculating the brightness similarity value, the contrast similarity value and the structural similarity value of the current cooking image and the previous cooking storage image;

[0021] calculating the difference value of the current cooking image and the previous cooking storage image based on the product of the brightness similarity value, the contrast similarity value and the structural similarity value of the current cooking image and the previous cooking storage image.

[0022] The application calculates the product of the brightness similarity value, the contrast similarity value and the structural similarity value of the current cooking image and the previous cooking storage image respectively, determines the difference value of the current cooking image and the previous cooking storage image, compares the image brightness, contrast and structure from multiple aspects, ensures the accuracy of the final difference value calculation result, and provides accurate data basis for subsequent selection and storage of the cooking video.

[0023] In an optional embodiment, the calculation of the brightness similarity value, the contrast similarity value and the structural similarity value of the current cooking image and the previous cooking storage image comprises:

[0024] graying the current cooking image and the previous cooking storage image respectively;

[0025] respectively calculate the pixel mean, pixel variance and pixel covariance of the current cooking image and the last cooking stored image after the gray processing;

[0026] based on the pixel mean and the first preset stability coefficient, calculate the brightness similarity value of the current cooking image and the last cooking stored image after the gray processing;

[0027] based on the pixel variance and the second preset stability coefficient, calculate the contrast similarity value of the current cooking image and the last cooking stored image after the gray processing;

[0028] based on the pixel covariance and the third preset stability coefficient, calculate the structural similarity value of the current cooking image and the last cooking stored image after the gray processing.

[0029] The present application provides accurate data basis for the selection and storage of subsequent cooking videos by calculating the brightness similarity value, the contrast similarity value and the structural similarity value respectively by using the pixel mean, the pixel variance and the pixel covariance.

[0030] In an optional embodiment, the first cooking image is extracted from the cooking images recorded in the current cooking video recording period for storage, comprising:

[0031] based on the proportion relationship between the current cooking video recording period and the preset maximum cooking video recording period and the preset first maximum extraction quantity threshold, determine the extraction quantity of the first cooking image in the current cooking video recording period;

[0032] based on the extraction quantity, extract the first cooking image from the cooking images recorded in the current cooking video recording period for storage.

[0033] The present application sets the preset first maximum extraction quantity threshold and determines the number of cooking images to be stored in the current cooking video recording period in combination with the proportion of the current cooking video recording period and the preset maximum cooking video recording period, and extracts and stores the cooking images, so that the longer the current cooking video recording period is, the more cooking images are stored, thereby balancing the cooking video recording time length and the demand of the final cooking video for cooking images to users, and improving the user experience.

[0034] In an optional embodiment, the second cooking image is extracted from the cooking images recorded in the current cooking video recording period for storage, comprising:

[0035] based on the preset second maximum extraction quantity threshold, extract the second cooking image from the cooking images recorded in the current cooking video recording period for storage.

[0036] The application stores the second cooking image according to a fixed number, limits the storage space required by the cooking video, controls the product cost, and improves the user experience on the basis of ensuring that the cooking video completely shows the cooking process.

[0037] In a second aspect, the application provides a cooking video processing device applied to a cooking device, and the device comprises:

[0038] A first processing module is configured to calculate a difference value of each frame of cooking image recorded by the cooking device in a current cooking video recording period and a last cooking storage image, respectively.

[0039] A second processing module is configured to determine whether the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the last cooking storage image is greater than a preset difference value threshold.

[0040] A third processing module is configured to extract a first cooking image from the cooking image recorded in the current cooking video recording period and store the first cooking image when the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the last cooking storage image is greater than the preset difference value threshold.

[0041] A fourth processing module is configured to splice all cooking images stored by the cooking device in the entire cooking process according to the storage time to obtain a target cooking video.

[0042] In a third aspect, the application provides a cooking device, which comprises an image acquisition device and a control unit, the image acquisition device is configured to acquire cooking images of the cooking device in a cooking process and send the acquired cooking images to the control unit, and the control unit comprises:

[0043] A memory and a processor, which are in communication connection with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the cooking video processing method of the first aspect or any of the corresponding embodiments thereof.

[0044] In an optional embodiment, the cooking device is a steaming oven or an oven.

[0045] In a fourth aspect, the application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make a computer execute the cooking video processing method of the first aspect or any of the corresponding embodiments thereof. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0047] Figure 1 is a flowchart of a cooking video processing method according to an embodiment of the present application;

[0048] Figure 2 is a flowchart of another cooking video processing method according to an embodiment of the present application;

[0049] Figure 3 is a schematic diagram of an algorithm principle for recording a cooking video;

[0050] Figure 4 is a structural block diagram of a cooking video processing device according to an embodiment of the present application;

[0051] Figure 5 is a structural diagram of a control unit of a cooking device according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0053] With the popularity of cooking devices such as steam ovens and the development of related technologies, their functions are becoming more and more rich, and users' demand for intelligentization is becoming more and more strong. Some steam ovens are equipped with a camera module, which can record a video of the whole cooking process. Usually, a cooking process lasts at least ten minutes, and the working time of most intelligent menus is about 30 minutes. This results in that the original video recorded by the camera is long in time, and after recording the whole cooking process, the video file will be very large. If this function needs to be met, the camera module needs to be configured with a very large storage. In view of the cost, it is necessary to consider reducing the size of the saved video image, reducing the demand for storage, and reducing the design cost.

[0054] In the related art, the following mainstream technologies are used to reduce the file size of a video: selecting a suitable encoder, adjusting the resolution and frame rate, code rate control, compression format selection, etc. The above technologies can reduce the size of the recorded video to some extent, but on the basis of meeting the basic definition, the above technologies can only reduce the size of the video file to some extent, and the finally generated video is still large, and if it is compressed too much, it will not be able to guarantee the special requirements of cooking videos that need to show the complete cooking process to the user.

[0055] Through analysis of video content and business requirements, it is found that there are too many similar images in the time dimension of cooking videos, and the business requirements do not require saving all images at all times. Therefore, the cooking video recorded during the video recording process can be processed according to this feature to improve the problem of large cooking recording video files.

[0056] According to an embodiment of the present application, a cooking video processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0057] In this embodiment, a cooking video processing method is provided, which can be used in the control unit of a cooking device, such as CPU, single-chip microcomputer, etc. Figure 1 The flowchart of the cooking video processing method according to an embodiment of the present application is shown in FIG. 1, which includes the following steps: Figure 1 As shown in FIG. 1, the flowchart includes the following steps:

[0058] Step S101, respectively calculating the difference value of each frame of cooking image recorded by the cooking device in the current cooking video recording period and the last cooking storage image.

[0059] Specifically, in the initial stage when the image acquisition device on the cooking device, such as the camera module, starts video acquisition, the cooking storage image is empty. At this time, the first frame of cooking image collected is stored, and then the difference value of each frame of cooking image recorded in the current cooking video recording period and the first frame of cooking image is calculated on the basis of the first frame of cooking image. The current cooking video recording period is the time length of the cooking image frame buffer queue of the cooking device. Since the image sampling of the image acquisition device is fixed interval, the current cooking video recording period can be determined by the cooking image acquisition time length, or by the cooking image acquisition quantity and the cooking image sampling period, which is not limited by the present application.

[0060] Further, the difference value is an evaluation parameter for indicating whether two images are similar. The larger the difference value, the greater the difference between the two images, and vice versa. If the difference value is smaller, it means that the two images are more similar.

[0061] Step S102, judge whether the mean value of the difference value between each frame of cooking image recorded in the current cooking video recording period and the last cooking stored image is greater than the preset difference value threshold.

[0062] The preset difference value threshold is the best value obtained by a large number of cooking video recording experiments, and in actual application, it can also be flexibly set according to the requirements of video recording size and the needs of video recording effect, and the application is not limited thereto.

[0063] Step S103, when the mean value of the difference value between each frame of cooking image recorded in the current cooking video recording period and the last cooking stored image is greater than the preset difference value threshold, the first cooking image is extracted from the cooking image recorded in the current cooking video recording period and stored.

[0064] Specifically, if the mean value of the difference value between each frame of cooking image recorded in the current cooking video recording period and the last cooking stored image is greater than the preset difference value threshold, it indicates that the cooking image has changed greatly in this recording time, in order to guarantee the integrity and continuity of the cooking process, the cooking image needs to be selected and stored to meet the display requirements of the cooking video.

[0065] Step S104, all cooking images stored by the cooking equipment in the entire cooking process are spliced according to the storage time to obtain the target cooking video.

[0066] Specifically, since the storage time of the cooking image is stored in sequence according to the recording process, all stored cooking images can be spliced and synthesized into the final cooking video in the order of storage time. The process of obtaining a video by image splicing is a prior art, which will not be described here.

[0067] In the embodiment of the application, the mean value of the difference value between the cooking image recorded in the current cooking video recording period and the last cooking stored image is compared with the preset difference value threshold, and when the difference value is greater than the preset difference value threshold, the cooking image is selected and stored from the current cooking video recording period. This cooking image selection method not only meets the user's requirement of showing the change of food materials in the cooking process, but also greatly reduces the number of stored cooking images through partial cooking image selection, thereby reducing the file size of the target cooking video obtained by splicing, and reducing the storage requirements of the cooking equipment, thereby reducing the production cost of the cooking equipment.

[0068] In the embodiment, a cooking video processing method is provided, which can be used for the control unit of the cooking equipment, such as CPU, single-chip microcomputer, etc. Figure 2is a flowchart of a cooking video processing method according to an embodiment of the present application, as shown in Figure 2 The flowchart includes the following steps:

[0069] In step S201, the difference value of each frame of cooking image recorded by the cooking device in the current cooking video recording period and the last cooking storage image is calculated respectively.

[0070] Specifically, the above step S201 includes:

[0071] In step S2011, the brightness similarity value, the contrast similarity value and the structural similarity value of the current cooking image and the last cooking storage image are calculated.

[0072] In step S2012, the difference value of the current cooking image and the last cooking storage image is calculated based on the product of the brightness similarity value, the contrast similarity value and the structural similarity value of the current cooking image and the last cooking storage image.

[0073] The embodiment of the present application determines the difference value of the current cooking image and the last cooking storage image by calculating the product of the brightness similarity value, the contrast similarity value and the structural similarity value of the current cooking image and the last cooking storage image respectively, so that the comparison can be made from the image brightness, contrast and structure, and the accuracy of the final difference value calculation result is guaranteed, and accurate data basis is provided for the subsequent selection and storage of the cooking video.

[0074] In some optional embodiments, the above step S2012 includes:

[0075] In step a1, the current cooking image and the last cooking storage image are respectively subjected to gray scale processing.

[0076] In step a2, the pixel mean value, the pixel variance and the pixel covariance of the current cooking image and the last cooking storage image after the gray scale processing are calculated respectively.

[0077] In step a3, the brightness similarity value of the current cooking image and the last cooking storage image is calculated based on the pixel mean value of the current cooking image and the last cooking storage image after the gray scale processing and the first preset stability coefficient.

[0078] In step a4, the contrast similarity value of the current cooking image and the last cooking storage image is calculated based on the pixel variance of the current cooking image and the last cooking storage image after the gray scale processing and the second preset stability coefficient.

[0079] In step a5, the structural similarity value of the current cooking image and the last cooking storage image is calculated based on the pixel covariance of the current cooking image and the last cooking storage image after the gray scale processing and the third preset stability coefficient.

[0080] The embodiment of the present application calculates the luminance similarity value, the contrast similarity value and the structure similarity value by using the pixel mean value, the pixel variance and the pixel covariance, respectively, to provide an accurate data basis for subsequent selection and storage of cooking videos.

[0081] Exemplarily, the difference value calculation process of the current cooking image and the last cooking stored image includes:

[0082] 1. First, perform gray processing on the two pictures, so that the value of each pixel of the picture is 0-255.

[0083] 2. Calculate the mean value mu_x, mu_y, the variance sigma_x, sigma_y and the covariance sigma_xy of the pixels of the two images x and y to be compared, respectively.

[0084] 3. Use the calculation results to calculate the luminance similarity, the contrast similarity and the structure similarity of the SSIM, respectively.

[0085] Luminance similarity: l=(2*mu_x*mu_y+C1) / (mu_x^2+mu_y^2+C1).

[0086] Contrast similarity: c=(2*sigma_x*sigma_y+C2) / (sigma_x^2+sigma_y^2+C2).

[0087] Structure similarity: s=(sigma_xy+C3) / (sigma_x*sigma_y+C3).

[0088] Similarity: SSIM=l*c*s.

[0089] Where C1, C2 and C3 are constants to prevent the denominator from being zero. Through the above process, the similarity of the two pictures can be roughly calculated. The result value of SSIM is from -1 to 1, and the smaller it is, the lower the similarity is. 1 represents complete identity. Based on 1, the difference value of the two pictures can be represented by the similarity.

[0090] In actual application, the algorithm can be made more sensitive by adjusting C1, C2 and C3. For different recipes, due to the different change ranges of the food cooking process, a smaller constant is used for a recipe with smaller changes, which can better capture the subtle differences of food changes; otherwise, a larger constant is used to make the result more stable.

[0091] In addition, in actual application, the image needs to be divided into regions for calculation in the process of calculating the difference value of two images. Since the cooking video is more inclined to capture the change of the food material, the similarity of the food material is calculated. Therefore, in the calculation process, the image is divided into regions, so that the weight of the calculation result of the region where the food material is located is higher, and the weight of other regions is lower. The similarity of each divided region is calculated, and then all the results are weighted and averaged to obtain the final result, so as to ensure that the final similarity calculation result can better reflect the similarity of the food material.

[0092] Step S202, judging whether the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the last cooking stored image is greater than the preset difference value threshold. For details, please refer to the related description of step S102 as shown in the figure Figure 1 The detailed description of step S102 is not repeated here.

[0093] Step S203, when the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the last cooking stored image is greater than the preset difference value threshold, extracting the first cooking image from the cooking image recorded in the current cooking video recording period for storage.

[0094] Specifically, the step S203 of extracting the first cooking image from the cooking image recorded in the current cooking video recording period for storage includes:

[0095] Step S2031, determining the extraction quantity of the first cooking image in the current cooking video recording period based on the proportional relationship between the current cooking video recording period and the preset maximum cooking video recording period and the preset first maximum extraction quantity threshold.

[0096] The first maximum extraction quantity threshold is less than the total amount of cooking video recorded in the preset maximum cooking video recording period. The first maximum extraction quantity threshold can be the best value obtained through a large number of recording experiments, or the size and effect of cooking video recording can be controlled by modifying the specific value, which is not limited in the present application.

[0097] Specifically, the extraction quantity can be obtained by multiplying the preset first maximum extraction quantity threshold and the above-mentioned proportional relationship. When the calculated extraction quantity is not an integer, if it is less than 1, the extraction quantity is determined as 1, and if it is not less than 1, the extraction quantity is determined by rounding.

[0098] Step S2032, extracting the first cooking image from the cooking image recorded in the current cooking video recording period based on the extraction quantity for storage.

[0099] Specifically, the last extraction quantity of cooking images recorded in the current cooking video recording period can be extracted and stored, or the maximum extraction quantity of cooking images recorded in the current cooking video recording period and different from the last cooking storage image can be extracted and stored, so as to guarantee the best display effect of the cooking images, but the present application is not limited thereto.

[0100] In the embodiment of the present application, the preset first maximum extraction quantity threshold is set, and the number of cooking images to be stored in the current cooking video recording period is determined in combination with the ratio of the current cooking video recording period to the preset maximum cooking video recording period, and the cooking images are extracted and stored, so that the longer the current cooking video recording period is, the more cooking images are stored, thereby taking into account the cooking video recording time length and the demand of the final cooking video for displaying cooking images to the user, and improving the user experience.

[0101] In step S204, when the average value of the difference values of each frame of cooking image recorded in the current cooking video recording period and the last cooking storage image is not greater than the preset difference value threshold, the next frame of cooking image recorded outside the current cooking video recording period is added to the current cooking video recording period, the current cooking video recording period is updated, and step S201 is returned.

[0102] For example, it is assumed that the current cooking video recording period is 50 seconds, the image acquisition device acquires a cooking image every 10 seconds, and 5 cooking images are recorded during this period. If the average value of the difference values of the 5 cooking images and the last cooking storage image is not greater than the preset difference value threshold, the 6th cooking image recorded is added to the current cooking video recording period, and the current cooking video recording period is updated to 60 seconds, thereby realizing real-time updating of the current cooking video recording period.

[0103] In the embodiment of the present application, when the average value of the difference values of each frame of cooking image recorded in the current cooking video recording period and the last cooking storage image is small, the cooking image is not stored, but the current cooking video recording period is continuously updated by continuously adding the newly recorded cooking image to the current cooking video recording period, and the average value of the difference values is compared again, thereby further guaranteeing that the finally stored cooking image is the cooking image with significant image change, further reducing the cooking video data volume while ensuring that the cooking video contains the key image with significant change of cooking materials, and improving the user experience.

[0104] In step S205, when the current cooking video recording period reaches the preset maximum cooking video recording period, the second cooking image is extracted from the cooking images recorded in the current cooking video recording period and stored.

[0105] Wherein, the preset maximum cooking video recording period is related to the total size of the cooking time, in general, the longer the total cooking time, the greater the time, and vice versa, which can also adjust the effect of video output. The food with longer cooking time usually changes slowly, and the change is not obvious in a period of time, so the size of the time needs to be adjusted according to the cooking time to ensure the effect of video output.

[0106] Specifically, the number of the second cooking images is less than the total number of the cooking images collected in the preset maximum cooking video recording period, and for example, only the last one or several cooking images recorded in the current cooking video recording period can be extracted and stored, or the one or several cooking images recorded in the current cooking video recording period and having the maximum difference value from the previous cooking stored image can be extracted and stored, so as to guarantee the best display effect of the cooking images, and the present application is not limited thereto.

[0107] In the present application, when the current cooking video recording period reaches the preset maximum cooking video recording period, and when the current video recording period is long, in order to accurately guarantee the integrity of the entire cooking process, the cooking images collected in the process are selected and stored, so as to avoid that the cooking video is oversimplified due to slow change of the food materials in some cooking processes, and further ensure the integrity of the cooking video, meet the actual needs of users for the cooking video, and improve the user experience.

[0108] Specifically, the second cooking images are extracted from the cooking images recorded in the current cooking video recording period based on a preset second maximum extraction number threshold.

[0109] Wherein, the second maximum extraction number threshold can be the best value obtained through a large number of recording experiments, or the size and effect of the cooking video recording can be controlled by modifying the specific value thereof, and the present application is not limited thereto.

[0110] In the present application, the second cooking images are extracted and stored according to the fixed number, so as to limit the storage space required by the cooking video, control the product cost, and improve the user experience on the basis of ensuring that the cooking video can completely display the cooking process.

[0111] Further, after the above step S205 or step S203 is performed, the cooking video processing method provided by the present application further comprises:

[0112] Step b1, deleting the cooking images recorded in the current cooking video recording period.

[0113] The embodiment of the present application reduces the data cache space required for recording the cooking video in the cooking process by deleting the cooking video recorded during the current cooking video recording period after storing the cooking video, further reduces the requirement for the data storage capacity of the cooking device, and reduces the cost.

[0114] In step S206, all cooking images stored by the cooking device during the entire cooking process are spliced according to the storage time to obtain the target cooking video. For details, refer to the related description of step S104 as shown in Figure 1 , which will not be repeated here.

[0115] The working principle and specific working process of the cooking video processing method provided by the embodiment of the present application will be described in detail below in combination with a specific application example.

[0116] The embodiment of the present application takes a steaming oven as an example, the steaming oven is provided with a camera module and a control unit (hereinafter referred to as a master control), the cooking video processing method is controlled and realized by the control unit on the steaming oven, and the overall scheme implementation includes three steps of real-time image acquisition, algorithm-controlled image saving, and synthesis of a complete video.

[0117] Firstly, the camera module receives the instruction of the steaming oven master control to start the camera for previewing and starts the video recording work of the cooking process. Usually, the camera records 24 frames of images in 1 second, but due to the application scene of the steaming oven and the computing power of the camera module, about one frame of image is taken per second in this scheme, which is used for subsequent algorithm processing and image saving.

[0118] The second step is to process the image frames generated by the camera in real time according to the algorithm, as shown in Figure 3 . The camera generates image frames in real time, which are input to the temporary storage queue of the program; the image frames in the judgment interval of the temporary storage queue are calculated and processed, and after meeting the conditions, some image frames in the temporary storage queue are saved to the save queue, which is a complete image saving process. The following is the specific condition of saving the image to the save queue.

[0119] Firstly, define the image at the end of the save queue as F pre , and the conditions for saving the image to the save queue are divided into two kinds. Condition one: the average difference value V e of the image in the judgment interval within the time T a > V t . The time T a represents the maximum time from the addition time of the image F pre at the end of the save queue to the current time, which can also represent the time when the temporary storage queue is at the maximum value, if it exceeds T a , then enter condition two.

[0120] average difference value V e representing the average of the difference values of each picture in the judgment interval and F pre , the use of the judgment interval is used to reduce the judgment error and reduce the impact of the sudden picture. The calculation of the similarity between the two images adopts the method of structural similarity index (SSIM), which quantifies the similarity between the two pictures from three aspects of brightness, contrast and structure. The result value of SSIM is from-1 to 1, the smaller the value is, the lower the similarity is, and 1 represents complete identity. Based on 1, the difference value between the two pictures can be confirmed. The difference value of each picture in the judgment interval and F pre e t representing the average difference value V e , greater than the threshold value, then the condition of triggering the picture saving is met. After meeting the condition of saving the picture, N images are extracted from the temporary queue and saved to the saving queue, wherein T c represents the time when condition one is met, that is, the time when F pre is saved to the current time when the condition is met, and M is a constant.

[0121] Condition two: when there is no image saving triggered in the time T a , condition two is entered. After entering condition two, the saving of the picture is triggered, and M2 frames of images are extracted from the temporary queue to the saving queue.

[0122] After saving triggered by condition one or condition two, the temporary queue is emptied, and F pre is updated. Wherein V t , M and M2 are constants, by modifying them, the size and effect of video recording can be controlled, and the final value is the best value obtained through a large number of recording experiments.

[0123] Finally, after the images in the saving queue are synthesized in time sequence, the final video can be generated.

[0124] Thus, by comparing the change rate of the food material image and saving the image data intelligently in real time according to the change rate, the problem of too large video file is solved.

[0125] In the embodiment, a cooking video processing device is also provided, which is used to realize the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that realizes a predetermined function. Although the device described in the following embodiments is preferably realized in software, the realization of hardware, or a combination of software and hardware, is also possible and conceived.

[0126] ​​The embodiment provides a cooking video processing device, as shown in the figure. Figure 4 The cooking video processing device comprises:

[0127] A first processing module 401 is configured to calculate a difference value of each frame of cooking image recorded by the cooking device in a current cooking video recording period and a last cooking storage image respectively.

[0128] A second processing module 402 is configured to judge whether the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the last cooking storage image is greater than a preset difference value threshold.

[0129] A third processing module 403 is configured to extract a first cooking image from the cooking image recorded in the current cooking video recording period and store the first cooking image when the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the last cooking storage image is greater than the preset difference value threshold.

[0130] A fourth processing module 404 is configured to splice all the cooking images stored by the cooking device in the whole cooking process according to the storage time to obtain a target cooking video.

[0131] In some optional embodiments, the cooking video processing device further comprises:

[0132] A fifth processing module is configured to add a next frame of cooking image recorded outside the current cooking video recording period into the current cooking video recording period, update the current cooking video recording period, and return to the step of calculating the difference value of each frame of cooking image recorded by the cooking device in the current cooking video recording period and the last cooking storage image respectively when the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the last cooking storage image is not greater than the preset difference value threshold.

[0133] In some optional embodiments, the cooking video processing device further comprises:

[0134] A sixth processing module is configured to extract a second cooking image from the cooking image recorded in the current cooking video recording period and store the second cooking image when the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the last cooking storage image is not greater than the preset difference value threshold and the current cooking video recording period reaches a preset maximum cooking video recording period.

[0135] In some optional embodiments, the cooking video processing device further comprises:

[0136] The seventh processing module is configured to delete the cooking images recorded in the current cooking video recording period after the first cooking image or the second cooking image is extracted from the cooking images recorded in the current cooking video recording period and stored.

[0137] In some optional embodiments, the first processing module 401 comprises:

[0138] The first processing unit is configured to calculate the luminance similarity value, the contrast similarity value and the structural similarity value of the current cooking image and the previous cooking stored image.

[0139] The second processing unit is configured to calculate the difference value of the current cooking image and the previous cooking stored image based on the product of the luminance similarity value, the contrast similarity value and the structural similarity value of the current cooking image and the previous cooking stored image.

[0140] In some optional embodiments, the second processing unit comprises:

[0141] The first processing sub-unit is configured to perform grayscale processing on the current cooking image and the previous cooking stored image respectively.

[0142] The second processing sub-unit is configured to calculate the pixel mean value, the pixel variance and the pixel covariance of the current cooking image and the previous cooking stored image after grayscale processing respectively.

[0143] The third processing sub-unit is configured to calculate the luminance similarity value of the current cooking image and the previous cooking stored image based on the pixel mean value of the current cooking image and the previous cooking stored image after grayscale processing and the first preset stability coefficient.

[0144] The fourth processing sub-unit is configured to calculate the contrast similarity value of the current cooking image and the previous cooking stored image based on the pixel variance of the current cooking image and the previous cooking stored image after grayscale processing and the second preset stability coefficient.

[0145] The fifth processing sub-unit is configured to calculate the structural similarity value of the current cooking image and the previous cooking stored image based on the pixel covariance of the current cooking image and the previous cooking stored image after grayscale processing and the third preset stability coefficient.

[0146] In some optional embodiments, the third processing module 403 comprises:

[0147] The third processing unit is configured to determine the number of the first cooking images extracted in the current cooking video recording period based on the proportional relationship between the current cooking video recording period and the preset maximum cooking video recording period and the preset first maximum extraction number threshold.

[0148] The fourth processing unit is configured to extract the first cooking image from the cooking images recorded in the current cooking video recording period based on the extraction quantity and store the first cooking image.

[0149] In some optional embodiments, the sixth processing module includes:

[0150] The fifth processing unit is configured to extract the second cooking image from the cooking images recorded in the current cooking video recording period based on a preset second maximum extraction quantity threshold and store the second cooking image.

[0151] Further function descriptions of the above-mentioned modules and units are the same as those of the corresponding embodiments, and will not be described here again.

[0152] The cooking video processing apparatus in the embodiment is presented in the form of functional units. The units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0153] The embodiment of the present application also provides a cooking device, which comprises an image acquisition device and a control unit. The image acquisition device is configured to acquire cooking images of the cooking device in a cooking process and send the acquired cooking images to the control unit. The control unit is configured to execute the method provided in the above-mentioned method embodiment. Exemplarily, the cooking device is a steam oven or an oven. In addition, in actual application, the cooking device can also be other cooking devices with a cooking video recording function, such as an air fryer, and the present application is not limited in this way.

[0154] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of the control unit of the cooking device provided in the optional embodiment of the present application, as Figure 5 shown, the control unit comprises one or more processors 10, a memory 20, and an interface for connecting various components, including a high-speed interface and a low-speed interface. Various components are communicatively connected to each other by different buses, and can be installed on a common mainboard or in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in the memory or the memory to display graphical information on the GUI of the external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, each device providing part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 In the above-mentioned control unit, the processor 10 is taken as an example.

[0155] The processor 10 can be a central processing unit, a network processing unit, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic device, a general array logic, or any combination thereof.

[0156] The memory 20 stores instructions executable by the at least one processor 10 to cause the at least one processor 10 to perform the methods illustrated in the above embodiments.

[0157] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system and application programs required by at least one function. The data storage area can store data created according to the use of the computer device, and the like. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some alternative embodiments, the memory 20 can optionally include a memory disposed remotely with respect to the processor 10, and these remote memories can be connected to the computer device through a network.

[0158] Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0159] The memory 20 can include a volatile memory, such as a random access memory, and can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid state disk. The memory 20 can further include a combination of the above-mentioned types of memories.

[0160] The control unit further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30, and the output device 40 can be connected by a bus or other means, Figure 5 For example, by way of a bus connection.

[0161] The input device 30 can receive input digital or character information, and generate key signal inputs related to the user settings and function controls of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, and the like. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), a tactile feedback device (e.g., a vibration motor), and the like. The display device includes, but is not limited to, a liquid crystal display, a light emitting diode, a display, and a plasma display. In some alternative embodiments, the display device can be a touch screen.

[0162] The embodiments of the present application further provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or be implemented as computer codes stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded through a network and stored in a local storage medium, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer codes, when the software or computer codes are accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0163] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A cooking video processing method applied to a cooking device, characterized in that, The method comprises: respectively calculating the difference value of each frame of cooking image recorded by the cooking device in the current cooking video recording period and the last cooking storage image; in the initial stage when the image acquisition device on the cooking device starts video acquisition, the cooking storage image is empty, at this time, the first frame of cooking image collected is stored, and then based on the first frame of cooking image, the difference value of each frame of cooking image recorded in the current cooking video recording period and the first frame of cooking image is calculated; determining whether the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the last cooking storage image is greater than a preset difference value threshold; when the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the last cooking storage image is greater than the preset difference value threshold, a first cooking image is extracted from the cooking image recorded in the current cooking video recording period and stored; all cooking images stored by the cooking device in the whole cooking process are spliced according to the storage time to obtain a target cooking video; when the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the last cooking storage image is not greater than the preset difference value threshold, the next frame of cooking image recorded outside the current cooking video recording period is added to the current cooking video recording period, the current cooking video recording period is updated, and the step of respectively calculating the difference value of each frame of cooking image recorded by the cooking device in the current cooking video recording period and the last cooking storage image is returned again; when the current cooking video recording period reaches the preset maximum cooking video recording period, a second cooking image is extracted from the cooking image recorded in the current cooking video recording period and stored.

2. The method of claim 1, wherein, After the first cooking image or the second cooking image is extracted from the cooking image recorded in the current cooking video recording period and stored, the method further comprises: deleting the cooking image recorded in the current cooking video recording period.

3. The method of claim 1, wherein, The calculation of the difference value of each frame of cooking image recorded by the cooking device in the current cooking video recording period and the last cooking storage image comprises: calculating the brightness similarity value, the contrast similarity value and the structural similarity value of the current cooking image and the last cooking storage image; based on the product of the brightness similarity value, the contrast similarity value and the structural similarity value of the current cooking image and the last cooking storage image, calculating the difference value of the current cooking image and the last cooking storage image.

4. The method of claim 3, wherein, The calculation of the brightness similarity value, the contrast similarity value and the structural similarity value of the current cooking image and the last cooking storage image comprises: respectively performing gray processing on the current cooking image and the last cooking storage image; respectively calculating the pixel mean value, the pixel variance and the pixel covariance of the gray-processed current cooking image and the last cooking storage image; based on the pixel mean value of the gray-processed current cooking image and the last cooking storage image and the first preset stability coefficient, calculating the brightness similarity value of the current cooking image and the last cooking storage image. The contrast similarity value of the current cooking image and the previous cooking storage image is calculated based on a pixel variance of the current cooking image after the grayscale processing and the previous cooking storage image and a second preset stability coefficient; The structural similarity value of the current cooking image and the previous cooking storage image is calculated based on a pixel covariance of the current cooking image after the grayscale processing and the previous cooking storage image and a third preset stability coefficient.

5. The method of claim 1, wherein, The first cooking image is extracted from the cooking images recorded in the current cooking video recording period and stored, including: The extraction number of the first cooking image in the current cooking video recording period is determined based on a proportional relationship between the current cooking video recording period and a preset maximum cooking video recording period and a preset first maximum extraction number threshold; The first cooking image is extracted from the cooking images recorded in the current cooking video recording period based on the extraction number and stored.

6. The method of claim 1, wherein, The second cooking image is extracted from the cooking images recorded in the current cooking video recording period and stored based on a preset second maximum extraction number threshold. The device includes:

7. A cooking video processing apparatus applied to a cooking device, characterized in that, The first processing module is configured to calculate a difference value of each frame of cooking image recorded in the current cooking video recording period and the previous cooking storage image respectively; in an initial stage when the image acquisition device on the cooking device starts video acquisition, the cooking storage image is empty, at this time, the first frame of cooking image acquired is stored, and then the difference value of each frame of cooking image recorded in the current cooking video recording period and the first frame of cooking image is calculated based on the first frame of cooking image; The second processing module is configured to judge whether the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the previous cooking storage image is greater than a preset difference value threshold; The third processing module is configured to store the first cooking image extracted from the cooking images recorded in the current cooking video recording period when the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the previous cooking storage image is greater than the preset difference value threshold; The fourth processing module is configured to splice all the cooking images stored by the cooking device in the whole cooking process according to the storage time to obtain a target cooking video; The fifth processing module is configured to add the next frame of cooking image recorded outside the current cooking video recording period into the current cooking video recording period when the average value of the difference value of each frame of cooking image recorded in the current cooking video recording period and the previous cooking storage image is not greater than the preset difference value threshold, update the current cooking video recording period, and return to the step of calculating the difference value of each frame of cooking image recorded in the current cooking video recording period and the previous cooking storage image respectively. ​ The sixth processing module is configured to: when a mean difference value of each frame of cooking image recorded in a current cooking video recording period and a last cooking storage image is not greater than a preset difference value threshold, and the current cooking video recording period reaches a preset maximum cooking video recording period, extract a second cooking image from the cooking images recorded in the current cooking video recording period and store the second cooking image.

8. A cooking apparatus, characterized by, The cooking video processing method comprises the following steps: An image acquisition device and a control unit, the image acquisition device is configured to acquire cooking images of a cooking device in a cooking process, and send the acquired cooking images to the control unit, the control unit comprises: A memory and a processor, which are in communication connection with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the cooking video processing method according to any one of claims 1-6.

9. The cooking apparatus according to claim 8, characterized in that, The cooking device is a steaming oven or an oven.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make the computer execute the cooking video processing method according to any one of claims 1-6.

Images