A method and device for detecting the kitchen environment, a storage medium, and an electronic device

Abstract

The present application discloses a method and device for detecting the kitchen environment, a storage medium and an electronic device, including: determining a static reference base image corresponding to the acquisition device according to a first set of kitchen environment images acquired by the acquisition devices arranged in the kitchen; extracting an image of a target area from the acquired second set of kitchen environment images according to the static reference base image; performing type prediction on the image of the target area as a candidate recognition image to determine a predicted type of an object in the target area; inputting the image of the target area as an image to be detected into a type detection model corresponding to the predicted type for detection to determine a safety state of the object in the kitchen environment; thereby avoiding consumption of computing resources and being able to provide the accuracy of recognition.

Description

[0001] This application is a divisional application of a Chinese patent application with an application date of January 16, 2024, an application number of 202410065645.7, and an invention title of "A Method and Device for Detecting the Back Kitchen Environment, a Storage Medium, and an Electronic Device". Technical Field

[0002] This application relates to the field of machine vision, and specifically to a method and device for detecting the back kitchen environment. This application also relates to a computer storage medium and an electronic device. Background Art

[0003] Open kitchens with bright stoves is a common safety and hygiene sign in kitchens. Initially, it meant keeping the light boxes in the kitchen bright during the cooking process so that the chef's working area could be clearly visible. This sign emphasizes the importance of food safety and hygiene, allowing customers to see the entire cooking process, increasing transparency and reliability. Through open kitchens with bright stoves, customers can observe the chef's operations with confidence, ensuring that the quality and hygiene of the food meet the standards.

[0004] Open kitchens with bright stoves is not just a sign; it is a concept related to the entire catering industry and food safety issues. The implementation of open kitchens with bright stoves can not only improve the public's awareness and trust in food safety and hygiene but also promote the management of catering enterprises and enhance service quality. In today's catering market, more and more restaurants and food ordering services have adopted the practice of open kitchens with bright stoves to improve the dining experience and satisfaction of customers.

[0005] With the continuous development of Internet technology, open kitchens with bright stoves have extended from a safety supervision process that allows customers to see the internal environment and operation process of the kitchen to the identification of image information collected by acquisition devices installed in the back kitchen (i.e., computer vision means) to detect potential safety hazards in real time, assisting merchants in improving safety management levels and hygiene standards, reducing food safety risks, and ensuring the quality and safety of food. In short, open kitchens with bright stoves play an important role in enhancing food safety awareness and service quality. Summary of the Invention

[0006] This application provides a method for detecting the back kitchen environment to solve the problem of excessive consumption of computing resources during the detection process in the prior art.

[0007] This application provides a method for detecting the back kitchen environment, including:

[0008] Determining a static reference base image corresponding to the acquisition device according to a first set of back kitchen environment images collected by acquisition devices arranged in the back kitchen;

[0009] Extract the image of the target area from the acquired second back-kitchen environment image set according to the static reference base image; use the image of the target area as a candidate recognition image for type prediction to determine the predicted type of the object in the target area;

[0010] Use the image of the target area as the image to be detected and input it into the type detection model corresponding to the predicted type for detection to determine the safety status of the object in the back-kitchen environment.

[0011] In some embodiments, determining the static reference base image corresponding to the acquisition device according to the first back-kitchen environment image set acquired by the acquisition device arranged according to the back-kitchen environment includes:

[0012] Perform object recognition on the images in the first back-kitchen environment image set to obtain recognition images;

[0013] Determine whether there are different images among the recognition images according to the comparison between the recognition images;

[0014] If not, determine the image randomly selected from the first back-kitchen environment image set as the static reference base image.

[0015] In some embodiments, performing object recognition on the images in the first back-kitchen environment image set to obtain recognition images includes:

[0016] Perform object edge detection on the images in the first back-kitchen environment image set to determine the object segmentation area in the image;

[0017] Perform object recognition on the segmentation area to obtain recognition images corresponding to the segmentation area.

[0018] In some embodiments, it further includes:

[0019] When it is determined that there are different images among the recognition images, determine whether the difference is within the difference threshold range determined according to the acquisition device specifications and / or acquisition angles;

[0020] If so, determine the image randomly selected from the first back-kitchen environment image as the static reference base image.

[0021] In some embodiments, it further includes:

[0022] When it is determined that the difference is not within the difference threshold range determined according to the acquisition device specifications and / or acquisition angles, determine the image with the least number of objects in the recognition images as the static reference base image corresponding to the acquisition device.

[0023] In some embodiments, determining a static reference base image corresponding to the acquisition device according to the first set of kitchen environment images acquired by the acquisition devices arranged in the kitchen includes:

[0024] Randomly select an image from the first set of kitchen environment images as a candidate static reference base image;

[0025] Perform object recognition on the images in the first set of kitchen environment images;

[0026] Update the candidate static reference base image according to the images in the first set of kitchen environment images where the number of recognized object objects is less than the number of object objects in the candidate static reference base image;

[0027] Determine whether the number of object objects in the candidate static reference base image is the image with the fewest object objects in the first set of kitchen environment images. If so, determine the updated candidate static reference base image as the static reference base image.

[0028] In some embodiments, taking the image of the target area as a candidate recognition image, predicting the type of the object object corresponding to the target area, and determining the predicted type of the object object in the target area includes:

[0029] Determine whether the area ratio between the target area and the static reference base image is greater than or equal to a preset threshold range;

[0030] If so, take the image of the target area as a candidate recognition image and input it into the type prediction model for type prediction;

[0031] Determine the predicted type of the object object in the target area according to the prediction result.

[0032] In some embodiments, taking the image of the target area as a candidate recognition image for type prediction and determining the predicted type of the object object corresponding to the target area includes:

[0033] Extract the feature data of the object object in the candidate recognition image according to the feature extraction method corresponding to the acquisition time of the candidate recognition image;

[0034] Perform type prediction according to the feature data to obtain the predicted type of the object object corresponding to the target area.

[0035] In some embodiments, the extracting the feature data of the object object in the candidate recognition image according to the feature extraction method corresponding to the acquisition time of the candidate recognition image includes:

[0036] When the acquisition time is within the daytime time period range, local feature data of the object in the candidate recognition image is extracted according to the local feature extraction method;

[0037] When the acquisition time is within the nighttime time period range, specular reflection feature data of the object in the candidate recognition image is extracted according to the specular reflection feature extraction method.

[0038] In some embodiments, determining a static reference base image corresponding to the acquisition device according to the first set of images of the back kitchen environment acquired by the acquisition devices arranged in the back kitchen includes:

[0039] According to the acquisition dimension configured for the set of images of the back kitchen environment, the first set of images of the back kitchen environment of the acquisition device is acquired; wherein, the acquisition dimension includes at least one of: time dimension, food type dimension, back kitchen location dimension;

[0040] According to the first set of images of the back kitchen environment, a static reference base image corresponding to the acquisition device is determined.

[0041] This application also provides a back kitchen environment detection device, including:

[0042] A first determination unit, configured to determine a static reference base image corresponding to the acquisition device according to the first set of images of the back kitchen environment acquired by the acquisition devices arranged in the back kitchen;

[0043] An extraction unit, configured to extract an image of a target area from the acquired second set of images of the back kitchen environment according to the static reference base image;

[0044] A second determination unit, configured to perform type prediction on the image of the target area as a candidate recognition image, and determine the predicted type of the object corresponding to the target area;

[0045] A third determination unit, configured to input the image of the target area as a to-be-detected image into a type detection model corresponding to the predicted type of the object for detection, and determine the safety state of the object in the back kitchen environment.

[0046] This application also provides a computer storage medium for storing a computer program;

[0047] The program executes the above-mentioned back kitchen environment detection method.

[0048] This application also provides an electronic device, including:

[0049] A processor;

[0050] A memory for storing a computer program, and the program executes the above-mentioned back kitchen environment detection method.

[0051] Compared with the prior art, the present application has the following advantages:

[0052] A kitchen environment detection method provided by the present application first screens the images in the acquired first set of kitchen environment images to determine a static reference base image, and then compares the static reference base image with the images in the acquired second set of kitchen environment images to extract the images with target regions, thereby reducing the consumption of computing resources for subsequent prediction of the object type in the target region. Because, on the one hand, there is no need to globally identify the second set of kitchen environment images, and only by comparing it with the static reference base image can it be determined whether there is a target region; on the other hand, for type prediction, it is only for the extracted target region part rather than the whole image, so there is no need to consume a large amount of computing resources. Based on the type prediction result of the object corresponding to the target region, type detection is performed again, so that not only the accuracy of detecting the object type in the target region can be ensured, but also the accuracy of identifying the safety state in the kitchen environment can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flowchart of an embodiment of a kitchen environment detection method provided by the present application;

[0054] Figure 2 is a schematic diagram of object edge segmentation in an embodiment of a kitchen environment detection method provided by the present application;

[0055] Figure 3 is a schematic structural diagram of an embodiment of a kitchen environment detection device provided by the present application;

[0056] Figure 4 is a schematic structural diagram of an embodiment of an electronic device provided by the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] Many specific details are set forth in the following description in order to provide a thorough understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the connotation of the present application. Therefore, the present application is not limited by the specific embodiments disclosed below.

[0058] The terms used in the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The descriptive methods used in the present application and the appended claims, such as "a", "first", and "second", etc., are not intended to limit the quantity or the order, but are used to distinguish the same type of information from each other.

[0059] According to the above background art, the inventive concept of this application comes from the application scenario of "transparent kitchens and bright dining areas". Specifically, in the application scenario of "transparent kitchens and bright dining areas", in the prior art, cameras are installed in the back kitchens of catering businesses to detect the operation norms of kitchen staff and the conditions of the kitchen environment. However, in the prior art, on the one hand, since cameras are provided by different manufacturers, abnormal situation recognition is performed locally through the cameras. This method is limited by the performance and hardware conditions of the cameras themselves, and different camera specifications have different image ratios. In the face of a large amount of collected data, refined recognition cannot be achieved due to performance reasons. On the other hand, because the hardware and manufacturers of the cameras are different, the local recognition capabilities are also different, resulting in inconsistent recognition standards and thus poor recognition consistency. Therefore, this application provides a method for detecting the kitchen environment to avoid the problems of low detection accuracy and large computational complexity caused by camera performance and hardware issues. The following describes a method for detecting the kitchen environment provided by this application.

[0060] As Figure 1 shown, Figure 1 is a flowchart of an embodiment of a method for detecting the kitchen environment provided by this application. The method includes:

[0061] Step S101: Determine a static reference base image corresponding to the acquisition device according to a first set of kitchen environment images acquired by the acquisition device arranged in the kitchen.

[0062] Step S102: Extract an image of the target area from the acquired second set of kitchen environment images according to the static reference base image.

[0063] Step S103: Use the image of the target area as a candidate recognition image for type prediction to determine the predicted type of the object corresponding to the target area.

[0064] Step S104: Input the image of the target area as an image to be detected into a type detection model corresponding to the predicted type of the object for detection to determine the safety state of the object in the kitchen environment.

[0065] The above steps S101 to S104 are described in detail below with specific examples.

[0066] Regarding step S101: Determine a static reference base image corresponding to the acquisition device according to a first set of kitchen environment images acquired by the acquisition device arranged in the kitchen.

[0067] Among them, the acquisition device can be a device such as a camera for acquiring image information or video information. The first set of images of the back kitchen environment can include multiple consecutive images, for example: sequential frame images in video data; or multiple non-consecutive images, for example: images sent respectively according to the set acquisition time. The multiple consecutive images and non-consecutive images can also be understood as the differences existing in the time interval. For example: consecutive images can have a shorter time interval between images, such as one frame per second, while non-consecutive images can have a longer time interval between images, such as one image per five seconds. The images in the first set of images of the back kitchen environment can be video stream data acquired by a camera, and the video stream data can include frame images with different time intervals. Of course, it can also be pictures sent by the acquisition device according to the set sending time. The static reference base image can be understood in this embodiment as the reference base image determined from the first set of images of the back kitchen environment.

[0068] The purpose of step S101 is to determine the static reference base image from the first set of images of the back kitchen environment. Since different acquisition devices have different specification signals and layout angles, when multiple acquisition devices are arranged in the back kitchen environment, it is necessary to establish corresponding static reference base images for different acquisition devices.

[0069] In this embodiment, when the acquisition device acquires the back kitchen environment information, it can be acquired according to different acquisition dimensions, specifically including:

[0070] Step S101-1: Obtain the first set of images of the back kitchen environment from the acquisition device according to the acquisition dimension configured for the set of images of the back kitchen environment; among them, the acquisition dimension includes at least one of the time dimension, the food type dimension, and the back kitchen location dimension;

[0071] Step S101-2: Determine the static reference base image corresponding to the acquisition device according to the first set of images of the back kitchen environment.

[0072] The specific implementation of step S101-1 may include: setting the acquisition frequency according to the high-incidence periods of food safety. For example, in summer, which is a high-incidence period of food safety, the acquisition frequency during this period can be increased to improve the detection frequency of the kitchen environment during this period. The acquisition frequency can also be set according to the high-incidence industries of food safety, that is, the high-incidence types of food safety. For example, the acquisition frequency can be increased for snack types such as spicy hot pot and barbecue to improve the detection frequency of the kitchen environment in this type of catering industry. The acquisition frequency can also be set according to the location of the merchant's kitchen. For example, if the merchant's kitchen is located on the street, the acquisition frequency can be increased; if the merchant's kitchen is located in a shopping mall and below the second floor of the shopping mall, the acquisition frequency can be increased. Of course, it can also include setting the acquisition frequency according to the number of kitchen staff, etc. The specific acquisition dimensions are not limited to the examples shown above and can be set in combination with the specific scenarios of food safety.

[0073] In this embodiment, it should be noted that the acquisition devices deployed in the kitchen environment can include various devices with different specifications. The providers of the acquisition devices can be different or the same. Whether the providers are different or the same, the image sets of the kitchen environment collected may have various formats. Therefore, adaptation processing is required to process the continuous or discontinuous images obtained from the acquisition devices into images that can be recognized by the CV large model or the multi-modal large model. The so-called CV large model refers to a large deep learning model for computer vision tasks, which is usually implemented using deep learning algorithms such as Convolutional Neural Network (CNN). In recent years, with the development of deep learning technology and the improvement of computing power, the CV large model has achieved many important results in the field of computer vision, such as image classification, object detection, image segmentation, etc. The basic idea of the CV large model is to learn the mapping from the input image to the output result and convert the input image into the output result, such as identifying the object category or location information in the image.

[0074] It is known from the prior art that the acquisition device is limited by its own performance, resulting in certain limitations in image processing, and thus the recognition result is not accurate enough. In this embodiment, to reduce the consumption of computing resources during anomaly detection, the processing efficiency is improved from the service side (or processing side) of image detection. Specifically, a static reference base map image that matches the acquisition device deployed in the kitchen environment is established, so as to reduce the calculation amount during the subsequent detection of kitchen safety risks. That is, taking the preliminary preparation as the focus to provide a reliable basis for subsequent detection, while improving the detection accuracy, the consumption of computing resources is also reduced. The specific implementation process can include various implementation methods, which will be described in turn below.

[0075] Method 1 includes:

[0076] Step S101-11: Perform object recognition on the images in the first back kitchen environment image set to obtain recognized images;

[0077] Step S101-12: Determine whether there are different images among the recognized images based on the comparison between the recognized images.

[0078] If not, randomly select an image from the first back kitchen environment image set and determine it as the static reference base image.

[0079] Among them, there can be at least two situations for the step S101-11 in this embodiment:

[0080] The first situation is when the first back kitchen environment image set is a continuous sequence of frame images in the video stream obtained by the acquisition device, object recognition can be performed according to the context between the images to obtain recognized images.

[0081] The second situation is when the first back kitchen environment image set is a non-continuous image obtained by the acquisition device, the object edges in the image can be determined by edge detection (as shown in Figure 2 ), and specifically, it can include:

[0082] Step S101-111: Perform object edge detection on the images in the first back kitchen environment image set to determine the object segmentation regions in the images;

[0083] Step S101-112: Perform object recognition on the segmentation regions to obtain recognized images corresponding to the segmentation regions. It can be to determine the contours of the objects in the segmentation regions through object recognition, and use the contours and the object objects within the contours as recognized images.

[0084] It should be noted that object recognition can be achieved for both continuous and non-continuous images through steps S101-111 and S101-112. To improve the detection efficiency, context-based recognition can be used for continuous images.

[0085] The purpose of the step S101-12 is to determine whether there are differences or changes between the recognized images, which can be through contour comparison and / or pixel value comparison, etc., without limitation.

[0086] In Method 1, object recognition is performed on the images in the first back kitchen environment image set, and then the recognized images are compared to determine whether there are differences between the images. If not, it means that there are no object changes or living things in the images of the first back kitchen environment image set. Therefore, a random image can be selected from the first back kitchen environment image set as the static reference base image. On the contrary, if there are differences, it means that there are object changes or living things in the images. In this case, further judgment is needed because the occurrence of differences is not necessarily due to the appearance of living things. For the back kitchen environment, object changes occur more frequently during food processing. For example, if the image is of the operation area, the possibility of image differences occurring is very high. Therefore, the possibility of changes occurring is related to the specifications and / or installation angles of the acquisition devices. For example, for the same area, the ratio of the selected image ranges is different for wide-angle lenses and fish-eye lenses. To improve the accuracy of judgment, it can also include:

[0087] When it is determined that there is a difference image with differences in the recognized images, determine whether the difference is within the difference threshold range determined according to the specifications of the acquisition device and / or the acquisition angle; wherein, the difference threshold range can be set according to the specifications of the acquisition device and / or the acquisition angle, and of course, it can also be determined according to historical experience values. For example, it can be between 5% and 30%.

[0088] If so, determine the image randomly selected from the first back kitchen environment images as the static reference base image.

[0089] If not, determine the image with the least number of object objects in the recognized images as the static reference base image corresponding to the acquisition device.

[0090] Method 2 includes:

[0091] Step S101-21: Randomly select an image from the first back kitchen environment image set as the candidate static reference base image;

[0092] Step S101-22: Perform object recognition on the images in the first back kitchen environment image set;

[0093] Step S101-23: Update the candidate static reference base image according to the images in the first back kitchen environment image set whose recognized number of object objects is less than the number of object objects in the candidate static reference base image;

[0094] Step S101-24: Determine whether the number of object objects in the candidate static reference base image is the image with the fewest object objects in the first back kitchen environment image set. If so, determine the updated candidate static reference base image as the static reference base image.

[0095] For the object recognition in Method 2, it can also be performed in the same way as in Method 1.

[0096] Regardless of whether it is Method 1 or Method 2, the static reference base image can be the image with the fewest objects in the image. When comparing by identifying images, it can be iteratively updated based on the number of objects in the image. After comparing all the images in the first back kitchen environment image set, the static reference base image is determined. The purpose of Method 1 is to determine whether there are differences in the identified images by comparing the identified images. When there are no differences in the identified images, it is determined as the static reference base image. Method 2 mainly determines the image with the fewest objects by comparing the number of objects in the randomly selected candidate static reference base images, and then updates the candidate static reference base image.

[0097] The above is the determination of the static reference base image. It can be understood that the above determination process can be applied to different first back kitchen environment image sets. For example, the images in the first back kitchen environment image set are continuous images or discontinuous images. For the determination of the static reference base image in continuous images, the following method can also be used:

[0098] Extract the first frame image in the sequence frame images as the reference image;

[0099] According to the order of the sequence frame images, compare the second frame image with the first frame image and extract the different parts;

[0100] Perform object recognition on the different parts. If there is a target object, use the part outside the target object area as the static reference base image; repeat this process until all the continuous images are recognized. The areas outside the areas where the target objects are recognized can be saved as the static reference base image. It can be understood that the determination of the static reference base image for continuous images can be composed of the local splicing of pictures selected at different times, which is faster than identifying the contours and areas for discontinuous pictures.

[0101] After determining the static reference base image through step S101, according to the obtained second back kitchen environment image set and the static reference base image, extract the images in the second back kitchen environment image set that have target areas. That is, execute step S102.

[0102] Regarding step S102: Extract the image of the target area from the obtained second back kitchen environment image set according to the static reference base image.

[0103] The specific implementation process of the step S102 can be to compare the images in the second back kitchen environment image set with the static reference base image, and extract the images with the target area in the images, so that it is not necessary to identify the objects in the images in the second back kitchen environment image set one by one, and only need to compare them with the static reference base image, reducing the overhead of computing resources.

[0104] It should be noted that the purpose of this embodiment is to detect the safety problems existing in the back kitchen environment to improve food safety. Therefore, the extraction of the target area can be determined in combination with the relevant objects related to safety problems. For example, there is no one in the static reference base image. When comparing it with the images in the second back kitchen environment image set, if there is someone, the relevant area of the person is extracted. Another example is the extraction of still objects related to safety problems. If there is a trash can in the static reference base image and the images in the second back kitchen environment image set, the trash can is used as the target area for extraction, so as to detect the safety status of the trash can subsequently, such as whether the lid is closed and whether the garbage overflows. Of course, if the target image to be extracted is determined based on changes, the static reference base image can be a base image in a safe state. In this embodiment, it is also described that the number of objects included in the static reference base image is the image with the least number in the obtained first back kitchen environment image set, and it can include images of still objects in a safe state, such as the closed state of the trash can. Of course, if it is in the open state, it does not affect the subsequent extraction of the target area and the subsequent type prediction and safety status recognition.

[0105] Regarding step S103: Use the image of the target area as a candidate recognition image for type prediction to determine the predicted type of the object in the target area.

[0106] To improve the accuracy of the detection result, the target area can be further determined, which can specifically include:

[0107] Step S103-11: Determine whether the area ratio of the target area in the static reference base image is greater than or equal to a preset threshold range. For example, assume the target area is a reflective point, and there is a certain gap between its area ratio in the static reference base image and the area ratio of the reflective point of a living creature's eye in the static reference base image. Assume the target area is the contour of a mouse, and its area proportion in the static reference base image is necessarily smaller than the area proportion of the contour of a human body as the target area in the static reference base image. That is to say, to avoid misidentifying the light-emitting point of other devices as the eye of a living creature and thus not necessarily performing identification calculations and other processes using computing resources, it is possible to determine whether type prediction is required through further threshold range judgment. The preset threshold range can be a general preset empirical value or based on the data involved in the process of the current camera determining the static reference base image, such as the data involved in the iterative optimization of the static reference base image. For example, if a changing area of 5% - 7% may be a mouse, but the camera is a fish-eye lens, there will be a certain degree of magnification and distortion. Therefore, the preset threshold range can also be adjusted. After adjustment, for example, it can be 3% - 6%. Therefore, the preset threshold range can be adjusted in real time according to factors such as the specifications, performance, and installation angle of the camera.

[0108] Step S103-12: If so, use the image of the target area as a candidate recognition image and input it into the type prediction model for type prediction.

[0109] Step S103-13: Determine the predicted type of the object in the target area according to the prediction result.

[0110] The specific recognition process of step S103 may include:

[0111] Step S103-21: Extract the feature data of the object in the candidate recognition image according to the feature extraction method corresponding to the acquisition time of the candidate recognition image.

[0112] Step S103-22: Perform type prediction according to the feature data to obtain the predicted type of the object corresponding to the target area.

[0113] The acquisition time in step S103-21 can be a daytime time period or a nighttime time period. Usually, the probability of living creatures such as cats and mice appearing in the back kitchen at night is relatively high. At this time, feature extraction can be performed through light reflection. Daytime usually belongs to working hours, mainly for detecting the behavior norms of back kitchen staff and equipment such as trash cans. Therefore, a local feature extraction method can be used. Of course, whether it is a daytime time period or a nighttime time period, it is not limited to these two feature extraction methods, and the local feature extraction method can also be applied to nighttime feature extraction.

[0114] Therefore, the specific implementation process of step S103-21 may include:

[0115] Step S103-21-1: When the acquisition time is within the daytime time period range, extract the local feature data of the object in the candidate recognition image according to the local feature extraction method;

[0116] Step S103-21-2: When the acquisition time is within the nighttime time period range, extract the specular reflection feature data of the object in the candidate recognition image according to the light reflection feature extraction method.

[0117] Regarding step S104: Input the image of the target area as the image to be detected into the type detection model corresponding to the prediction type for detection, and determine the safety status of the object in the back kitchen environment. Specifically, when the predicted type of the object in the target area is a back kitchen staff member, input the image of the target area as the image to be detected into the first type detection model (a detection model for people) for detection, and determine whether the clothing of the back kitchen staff member complies with the regulations. If not, send an abnormal safety status message to the merchant; and / or, determine whether the back kitchen staff member has any violation behaviors. If so, send an abnormal safety status message to the merchant. When the object in the target area is a static object, such as a trash can, etc., input the image of the target area as the image to be detected into the second type detection model (a detection model for static objects) for detection, and determine whether the static object is in a safe state. For example: whether the trash can is covered, whether the gas stove is turned off, the usage status of the cutting board, etc. When the object in the target area is a living thing, such as a mouse, a cat, a dog, a cockroach, etc., input the image of the target area as the image to be detected into the third type detection model (a detection model for living things) for detection, and determine whether the living thing is in a safe state.

[0118] Of course, the above examples, such as static objects, living things, people, etc., are only examples in combination with conventional scenarios. Specifically, the back kitchen detection related to food safety may also include: objects that cause safety hazards in the back kitchen, such as: objects unrelated to the back kitchen (such as lighters, power banks that are prone to causing fires), the status of the gas stove in the back kitchen, the status inside the cookware, etc. Anything related to back kitchen safety and food safety can be used as the content that needs to be detected.

[0119] The above is a description of an embodiment of a method for detecting the kitchen environment provided by this application. In this method embodiment, first, images in the acquired first set of kitchen environment images are screened to determine a static reference base image. Then, the static reference base image is compared with the images in the acquired second set of kitchen environment images to extract the images with target regions, thereby reducing the consumption of computing resources for subsequent prediction of the object type in the target region. Because, on the one hand, there is no need to globally identify the second set of kitchen environment images, and only by comparing it with the static reference base image can it be determined whether there is a target region; on the other hand, for type prediction, it is only for the extracted target region part rather than the whole image. Therefore, a large amount of computing resources do not need to be consumed. Based on the type prediction result of the object corresponding to the target region, type detection is performed again, so that not only the accuracy of detecting the object type in the target region can be ensured, but also the accuracy of identifying the safety state in the kitchen environment can be ensured.

[0120] The above is a specific description of an embodiment of a method for detecting the kitchen environment provided by this application. Corresponding to the foregoing embodiment of a method for detecting the kitchen environment provided, this application also discloses an embodiment of a device for detecting the kitchen environment. Please refer to Figure 3 , since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For related parts, refer to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0121] As Figure 3 shown, Figure 3 is a schematic structural diagram of an embodiment of a device for detecting the kitchen environment provided by this application. This device embodiment may include:

[0122] A first determination unit 301, configured to determine a static reference base image corresponding to the acquisition device according to a first set of kitchen environment images acquired by an acquisition device arranged in the kitchen;

[0123] An extraction unit 302, configured to extract an image of a target region from a second set of kitchen environment images acquired according to the static reference base image;

[0124] A second determination unit 303, configured to use the image of the target region as a candidate recognition image for type prediction to determine a predicted type of an object corresponding to the target region;

[0125] A third determination unit 304, configured to use the image of the target region as a to-be-detected image and input it into a type detection model corresponding to the predicted type of the object to perform detection, and determine a safety state of the object in the kitchen environment.

[0126] In the first method of the first determination unit 301, it may include: an acquisition subunit, a first determination subunit, and a second determination subunit;

[0127] The acquisition subunit is configured to perform object recognition on the images in the first back kitchen environment image set to obtain recognition images;

[0128] The first determination subunit is configured to determine whether there are different images with differences in the recognition images according to the comparison between the recognition images;

[0129] When the determination result of the first determination subunit is negative, the second determination subunit is configured to determine the image randomly selected from the first back kitchen environment image set as the static reference base image.

[0130] The acquisition subunit may include: a region determination subunit and an image acquisition subunit;

[0131] The region determination subunit is configured to perform object edge detection on the images in the first back kitchen environment image set to determine the object segmentation region in the image;

[0132] The image acquisition subunit is configured to perform the object recognition on the segmentation region to obtain the recognition image corresponding to the segmentation region.

[0133] It further includes: a difference determination subunit and a selection subunit;

[0134] When it is determined that there are different images with differences in the recognition images, the difference determination subunit is configured to determine whether the difference is within the difference threshold range determined according to the acquisition device specifications and / or acquisition angles;

[0135] When the determination result of the difference determination subunit is positive, the selection subunit is configured to determine the image randomly selected from the first back kitchen environment images as the static reference base image; when the determination result of the difference determination subunit is negative, the selection subunit is configured to determine the image with the least number of object objects in the recognition images as the static reference base image corresponding to the acquisition device.

[0136] In the second method of the first determination unit 301, it may include: a selection subunit, an identification subunit, an update subunit, and a determination subunit;

[0137] The selection subunit is configured to randomly select an image from the first back kitchen environment image set as a candidate static reference base image;

[0138] The identification subunit is configured to perform object recognition on the images in the first back kitchen environment image set;

[0139] The updating subunit is configured to update the candidate static reference base image according to an image in the first set of kitchen environment images in which the number of object objects recognized is less than the number of object objects in the candidate static reference base image;

[0140] The determining subunit is configured to determine whether the number of object objects in the candidate static reference base image is the image with the fewest object objects in the first set of kitchen environment images. If so, the updated candidate static reference base image is determined as the static reference base image.

[0141] The first determining unit 301 may include: an obtaining subunit and a determining subunit;

[0142] The obtaining subunit is configured to obtain the first set of kitchen environment images of the acquisition device according to the acquisition dimension of the configured set of kitchen environment images; wherein, the acquisition dimension includes at least one of a time dimension, a meal type dimension, and a kitchen location dimension;

[0143] The determining subunit is configured to determine a static reference base image corresponding to the acquisition device according to the first set of kitchen environment images.

[0144] The second determining unit 303 may include: an area ratio determining subunit, a predicting subunit, and a determining subunit;

[0145] The area ratio determining subunit is configured to determine whether the area ratio between the target area and the static reference base image is greater than or equal to a preset threshold range;

[0146] When the determination result of the area ratio determining subunit is yes, the predicting subunit is configured to input the image of the target area as a candidate recognition image into a type prediction model for type prediction;

[0147] The determining subunit is configured to determine the predicted type of the object object in the target area according to the prediction result.

[0148] The second determining unit 303 may include: an extracting subunit and an obtaining subunit;

[0149] The extracting subunit is configured to extract the feature data of the object object in the candidate recognition image according to the feature extraction method corresponding to the acquisition time of the candidate recognition image; specifically, it may include: when the acquisition time is in the daytime time period interval, extracting the local feature data of the object object in the candidate recognition image according to the local feature extraction method; when the acquisition time is in the nighttime time period interval, extracting the specular reflection feature data of the object object in the candidate recognition image according to the specular reflection feature extraction method.

[0150] The obtaining subunit is configured to perform type prediction according to the feature data, and obtain the predicted type of the object corresponding to the target area.

[0151] For the content of the above device embodiment, reference may be made to steps S101 to S103 in the above method embodiment, which will not be elaborated here.

[0152] Based on the above content, the present application further provides a computer storage medium for storing a computer program;

[0153] The program executes the content of steps S101 to S103 involved in the above method embodiment of kitchen environment detection.

[0154] Based on the above content, the present application further provides an electronic device, as Figure 4 shown, including:

[0155] A processor 401;

[0156] A memory 402 for storing a computer program, and the program executes the content of steps S101 to S103 involved in the above method embodiment of kitchen environment detection.

[0157] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties. And the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0158] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0159] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0160] 1. A computer-readable medium includes both permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. As defined herein, a computer-readable medium does not include transitory media such as modulated data signals and carrier waves.

[0161] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0162] Although the present application is disclosed above in preferred embodiments, it is not intended to limit the present application. Any person skilled in the art can make possible objectives and modifications without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application should be subject to the scope defined by the claims of the present application.

Claims

1. A kitchen environment detection method, characterized in that: include: According to a first back kitchen environment image set collected by a collection device arranged in the back kitchen, a static reference base image corresponding to the collection device is selected; including: performing object recognition on the images in the first back kitchen environment image set to obtain a recognition image; determining whether there is a difference image with a difference in the recognition image according to a comparison between the recognition images; if not, determining an image randomly selected from the first back kitchen environment image set as the static reference base image; Extracting an image of a target area from a second acquired kitchen environment image set according to the static reference base image; Performing type prediction on the image of the target area as a candidate recognition image to determine the predicted type of the object in the target area; Inputting the image of the target area as the image to be detected into a type detection model corresponding to the predicted type for detection, and determining the safety status of the object in the back kitchen environment, including: When the predicted type of the object in the target area is a kitchen staff, the image of the target area is input as the image to be detected into the first type detection model for detection to determine whether the kitchen staff's attire complies with regulations; if not, send security status abnormality information to the merchant; and / or determine whether the kitchen staff has violated regulations, and if so, send security status abnormality information to the merchant; When the object in the target area is a still object, inputting the image of the target area as an image to be detected into the second type detection model for detection to determine whether the still object is in a safe state; When the object in the target area is a living creature, the image of the target area is input as an image to be detected into the third type detection model for detection to determine whether the living creature is in a safe state.

2. The kitchen environment detection method according to claim 1, characterized in that: The performing object recognition on the images in the first back kitchen environment image set to obtain a recognition image includes: Performing object edge detection on the images in the first kitchen environment image set to determine object segmentation areas in the images; The object recognition is performed on the segmented area to obtain a recognition image corresponding to the segmented area.

3. The kitchen environment detection method according to claim 1, characterized in that: The performing object recognition on the images in the first back kitchen environment image set to obtain a recognition image includes: When the first kitchen environment image set is a continuous sequence of frame images in a video stream acquired by an acquisition device, object recognition is performed according to contexts between the images to acquire a recognition image; When the first kitchen environment image set is a non-continuous image acquired by a collection device, the edge of the object in the image is determined by edge detection to acquire a recognition image.

4. The kitchen environment detection method according to claim 1, characterized in that: Also includes: When the determination of whether there is a difference image with a difference in the identified image is yes, determining whether the difference is within a difference threshold range determined according to the acquisition device specification and / or acquisition angle; If so, an image randomly selected from the first kitchen environment image is determined as the static reference base image.

5. The kitchen environment detection method according to claim 4, characterized in that: Also includes: When the determination of whether the difference is within the difference threshold range determined according to the acquisition device specification and / or acquisition angle is no, the image with the least number of objects in the recognition image is determined as the static reference base image corresponding to the acquisition device.

6. The method for detecting kitchen environment according to claim 1, characterized in that: Also includes: The static reference base image includes an image of a static object in a safe state.

7. The kitchen environment detection method according to claim 1, characterized in that: The step of selecting a static reference base image corresponding to the first kitchen environment image set collected by the collection device arranged in the kitchen includes: Randomly select an image from the first kitchen environment image set as a candidate static reference base image; Performing object recognition on the images in the first kitchen environment image set; According to an image in which the number of objects recognized in the first kitchen environment image set is less than the number of objects in the candidate static reference base image, updating the candidate static reference base image; Determine whether the candidate static reference base image has the least number of objects in the first kitchen environment image set; if so, determine the updated candidate static reference base image as the static reference base image.

8. The kitchen environment detection method according to claim 1, characterized in that: The step of taking the image of the target area as a candidate recognition image, predicting the type of the object corresponding to the target area, and determining the predicted type of the object in the target area includes: Determining whether an area ratio between the target area and the static reference base image is greater than or equal to a preset threshold range; If yes, the image of the target area is used as a candidate recognition image and input into the type prediction model for type prediction; The object prediction type of the target area is determined according to the prediction result.

9. The method for detecting kitchen environment according to claim 1, characterized in that: The step of taking the image of the target area as a candidate recognition image for type prediction and determining the predicted type of the object corresponding to the target area includes: Extracting feature data of an object in the candidate recognition image according to a feature extraction method corresponding to the acquisition time of the candidate recognition image; A type prediction is performed based on the feature data to obtain a predicted type of the object corresponding to the target area.

10. The kitchen environment detection method according to claim 9, characterized in that: The extracting feature data of the object in the candidate recognition image according to the feature extraction method corresponding to the acquisition time of the candidate recognition image includes: When the acquisition time is during the daytime, extracting local feature data of the object in the candidate recognition image in a local feature extraction manner; When the acquisition time is a nighttime period, the reflective feature data of the object in the candidate recognition image is extracted in accordance with the light reflection feature extraction method.

11. The kitchen environment detection method according to claim 1, characterized in that: The step of extracting an image of a target area from a second back kitchen environment image set acquired according to the static reference base image comprises: The extraction of the target area is combined with the extraction of related objects related to security issues, including: when there is no person in the static reference base image, if there is a person when the static reference base image is compared with the image in the second back kitchen environment image set, then extracting the area related to the person; For still life extraction involving safety issues, if there is a trash can in the static reference base image and the image set of the second kitchen environment, the trash can is extracted as the target area.

12. The kitchen environment detection method according to claim 1, characterized in that: The step of selecting a static reference base image corresponding to the first kitchen environment image set collected by the collection device arranged in the kitchen includes: Acquire a first back kitchen environment image set of the acquisition device according to the configured acquisition dimension of the back kitchen environment image set; wherein the acquisition dimension includes: at least one of a time dimension, a food type dimension, and a back kitchen location dimension; According to the first kitchen environment image set, a static reference base image corresponding to the acquisition device is determined.

13. The kitchen environment detection method according to claim 1, characterized in that: Also includes: The continuous images or discontinuous images acquired from the acquisition device are processed into images that can be recognized by the CV large model or the multimodal large model.

14. A kitchen environment detection device, characterized in that: include: A first determination unit is used to select a static reference base image corresponding to a first back kitchen environment image set collected by a collection device arranged in the back kitchen, according to the first back kitchen environment image set collected by the collection device; including: performing object recognition on the images in the first back kitchen environment image set to obtain a recognition image; determining whether there is a difference image with a difference in the recognition image according to a comparison between the recognition images; if not, determining an image randomly selected from the first back kitchen environment image set as the static reference base image; An extraction unit, configured to extract an image of a target area from a second acquired back kitchen environment image set according to the static reference base image; A second determination unit, configured to perform type prediction on the image of the target area as a candidate recognition image, and determine a predicted type of an object corresponding to the target area; The third determination unit is used to input the image of the target area as the image to be detected into the type detection model corresponding to the predicted type of the object for detection, and determine the safety status of the object in the back kitchen environment, including: when the predicted type of the object in the target area is a back kitchen staff, input the image of the target area as the image to be detected into the first type detection model for detection, and determine whether the dress of the back kitchen staff complies with the regulations; if not, send the safety status abnormality information to the merchant; and / or determine whether the back kitchen staff has any violation, and if so, send the safety status abnormality information to the merchant; when the object in the target area is a still object, input the image of the target area as the image to be detected into the second type detection model for detection, and determine whether the still object is in a safe state; when the object in the target area is a living thing, input the image of the target area as the image to be detected into the third type detection model for detection, and determine whether the living thing is in a safe state.

15. A computer storage medium for storing a computer program; The program executes the back kitchen environment detection method as described in any one of claims 1-13 above.

16. An electronic device, comprising: processor; A memory for storing a computer program, wherein the program executes the back kitchen environment detection method as described in any one of claims 1 to 13.

Images