A cooking unattended reminder method and system based on visual recognition

Abstract

The present invention relates to the field of visual recognition technology, and in particular to a method and system for reminding cooking without human intervention based on visual recognition, the method comprising: obtaining video streams or image data in a kitchen through a camera; using computer vision technology to identify various cooking devices in the video streams or image data, and locating and classifying different cooking devices in new video streams or images; analyzing the working status of cooking devices by analyzing the behavior patterns of different cooking devices; identifying dishes being cooked by the cooking devices, and judging whether human guarding is required after matching the corresponding cooking process flow in combination with the working status of the cooking devices; and selecting a corresponding safety reminder scheme according to the judgment result of the previous step, the safety reminder scheme comprising an on-duty reminder scheme and a self-processing reminder scheme.

Description

Technical Field

[0001] The present invention relates to the technical field of visual recognition, and in particular to a cooking unattended reminder method and system based on visual recognition. Background Art

[0002] In short, visual recognition technology uses computer image processing and analysis methods to enable machines to have functions similar to human vision, and can automatically extract and process information from images or video sequences. With the rapid development of deep learning and artificial intelligence technology, visual recognition technology has made significant progress and is widely used in security monitoring, medical diagnosis, intelligent manufacturing and other fields.

[0003] In the field of cooking, the application prospects of visual recognition technology are broad and important. It can not only improve cooking efficiency, but also greatly enhance kitchen safety and reduce safety accidents caused by human negligence. Especially in some situations where professional cooks are needed, accidents often occur due to the absence of cooks, which makes non-professionals unable to operate cooking equipment in a timely and correct manner. Summary of the invention

[0004] The present invention collects video stream data in the kitchen and uses visual recognition technology to identify and lock on personnel related to the cooking of dishes. During the cooking process that requires supervision by personnel, the present invention monitors and predicts the movement trajectory of personnel. When the distance between them and the cooking equipment exceeds a certain range, a reminder signal is generated to remind the relevant personnel to return to the work area.

[0005] The technical solution proposed by the present invention is: a cooking unattended reminder method based on visual recognition, the method comprising:

[0006] The video stream or image data in the kitchen is obtained through the camera, and the internal environment parameters of the cooking equipment during cooking are obtained through the sensor;

[0007] Using computer vision techniques to identify various cooking devices in video streams or image data, and to locate and classify different cooking devices in new video streams or images;

[0008] Analyze the working status of cooking equipment by analyzing the behavior patterns of different cooking equipment;

[0009] Identify the dishes being cooked by the cooking equipment, and after matching the cooking process with the working status of the cooking equipment, determine whether human supervision is required;

[0010] A corresponding safety reminder scheme is selected according to the judgment result of the previous step, and the safety reminder scheme includes an on-duty reminder scheme and a self-processing reminder scheme.

[0011] Preferably, the method of using computer vision technology to identify various cooking devices in video streams or image data includes:

[0012] Obtain kitchen history video streams or image data from a database, and mark the types and locations of cooking equipment in the video streams or image data to form a sample data set;

[0013] Divide the sample data set into a training set and a test set;

[0014] Using the training set to train a preset convolutional neural network, the convolutional neural network is enabled to extract contours, edges, and texture features of different cooking equipment from historical kitchen video stream data or image data, and learn features of different cooking equipment;

[0015] The performance of the convolutional neural network is tested through the test set, and the convolutional neural network that passes the performance test is deployed into the system.

[0016] Preferably, the locating and classifying different cooking devices in the new video stream or image includes:

[0017] According to the preset acquisition frequency, the video stream data or image data of various cooking equipment in the kitchen is acquired in real time to form a real-time kitchen image data set;

[0018] The convolutional neural network analyzes the video stream data frame by frame, uses the region proposal network to generate candidate regions including cooking equipment, and then classifies and regresses the generated candidate regions to determine the category and location of each candidate region;

[0019] Label the category and location of each candidate area;

[0020] Analyze the newly acquired images or video streams frame by frame to identify the type and coordinate information of each cooking device;

[0021] Construct a two-element array of device type and coordinate information ,in, Respectively represent the type and coordinates of the cooking equipment.

[0022] Preferably, the analyzing the working status of the cooking equipment by analyzing the behavior patterns of different cooking equipment includes:

[0023] After the convolutional neural network identifies the type and coordinate information of each cooking device, it performs behavioral pattern recognition on the identified devices;

[0024] Analyze the behavior patterns of cooking equipment and identify the working status of cooking equipment; specifically:

[0025] Acquiring specific data that can reflect a behavior pattern of the cooking device, wherein the specific data includes thermal image data, sensor data, and audio data of the cooking device;

[0026] By analyzing the thermal image data of the cooking device, the heat distribution characteristics of the cooking device are identified, and according to the heat distribution characteristics of the cooking device, the working state of the cooking device is identified;

[0027] By analyzing the sensor data, the pressure distribution characteristics of the cooking device are obtained, and the working state of the cooking device is identified according to the pressure distribution characteristics of the cooking device;

[0028] By analyzing the audio data, the working noise distribution characteristics of the cooking equipment are obtained, and the working status of the cooking equipment is identified based on the working noise distribution characteristics of the cooking equipment.

[0029] Preferably, the identifying of the dish being cooked by the cooking device, combined with the working state of the cooking device, after matching the corresponding cooking process flow, determines whether human guarding is required, including:

[0030] Identify the cooking equipment in operation and obtain information about dishes in the cooking equipment;

[0031] Get the working status of the cooking equipment in operation;

[0032] Obtaining a cooking process that matches the dish information from a database, extracting multiple cooking parameters at multiple key nodes from the cooking process, and forming multiple cooking parameter time series; the cooking parameters include cooking equipment type, temperature, pressure, and cooking time;

[0033] Based on the relationship between dish types, cooking techniques and equipment types, determine whether the dishes require human supervision during the cooking time.

[0034] Preferably, judging whether a dish needs human supervision during the cooking time according to the association between the dish type, cooking process and equipment type includes:

[0035] Define cooking equipment types and construct cooking equipment type sequences ,in, Indicates cooking equipment;

[0036] Define dish types and construct dish type sequences ,in, Indicates Plant vegetables;

[0037] Define cooking methods and build cooking method sequences ,in, Indicates cooking methods;

[0038] Define the personnel guard rule dictionary, and randomly extract an element from the cooking equipment type sequence, dish type sequence, and cooking method sequence to form a comparison sequence ,in, , , ;

[0039] Compare the comparison sequence with the personnel guard rule dictionary to determine whether the personnel guard rule is met. If so, determine whether the personnel guard rule is met. Types of cooking equipment Types of cooking methods When serving this type of dish, staff supervision is required; otherwise, no staff supervision is required.

[0040] Preferably, the selecting a corresponding safety reminder scheme according to the judgment result of the previous step includes:

[0041] Obtain kitchen video stream data and divide the video stream into multiple frame images;

[0042] Analyze multiple frames of images frame by frame to obtain the movement trajectory and behavior pattern of kitchen staff;

[0043] Establish the association between the behavior pattern of kitchen staff and the working status of cooking equipment on the timeline, and calculate the degree of association.

[0044] Establish the association between the behavior pattern of kitchen staff and the cooking process, and calculate the second degree of association. Screen the kitchen staff according to the first and second degrees of association to identify the personnel related to the cooking of dishes.

[0045] When human guarding is required, the on-duty reminder plan is executed; when human guarding is not required, the self-processing reminder plan is executed.

[0046] Preferably, when personnel guarding is required, a duty reminder scheme is executed, including:

[0047] Obtain kitchen video stream data, identify and lock people related to cooking from the video stream data;

[0048] The optical flow method is used to analyze multiple consecutive frame images containing personnel information related to the cooking of dishes frame by frame to obtain the movement trajectory of the personnel related to the cooking of dishes;

[0049] Predict the next location of the staff related to the dish based on the acquired action trajectory. If the next location is more than a preset safety distance from the dish cooking equipment, send a reminder signal 1 to notify the staff related to the dish to return to the working range.

[0050] When no personnel guarding is required, the self-processing reminder scheme is executed, including:

[0051] Obtain kitchen video stream data, and identify cooking equipment related to cooking dishes from the video stream data;

[0052] Identify the working status of corresponding cooking equipment;

[0053] When the working status changes, a second reminder signal is output to notify the staff related to the cooking of the dish to determine whether the working status is normal.

[0054] The present invention also provides a visual recognition-based unattended cooking reminder system, wherein the system is used to execute the visual recognition-based unattended cooking reminder method.

[0055] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the visual recognition-based unmanned cooking reminder method.

[0056] Beneficial effects of the present invention:

[0057] 1. The present invention uses a camera to obtain video streams or static images in the kitchen in real time to ensure the real-time and accuracy of the data. This step provides basic data support for subsequent visual recognition and analysis. Deep learning models (such as YOLO, Faster R-CNN, etc.) are used to identify and locate various cooking equipment in video streams or images and classify them. This step ensures accurate identification of different equipment and lays the foundation for behavioral pattern analysis.

[0058] 2. The present invention uses time series analysis or motion recognition technology to analyze the behavior patterns of different cooking equipment and determine their working status. This includes the start, run and stop status of the equipment, which helps to determine whether the equipment is working properly. Combined with image processing technology and deep learning models, the dishes being cooked are identified, and the corresponding cooking process is matched according to the identified dish type. This step helps to determine whether human supervision is needed to ensure the safety and smooth progress of the cooking process. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 The present invention is a flow chart of a method for unmanned cooking reminder based on visual recognition. DETAILED DESCRIPTION

[0060] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0061] It is to be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" should not be understood as a limitation on the quantity.

[0062] refer to Figure 1 The technical solution provided by the present invention is: a cooking unattended reminder method based on visual recognition, comprising the following steps:

[0063] 1. Obtain video stream or image data in the kitchen through a camera.

[0064] In this embodiment, a camera is used to capture the video stream in the kitchen in real time, which can be implemented using the VideoCapture class in the OpenCV library. A static image can also be captured at a specific time point using the imread function of OpenCV. The camera captures various activities occurring in the kitchen, including the use of cooking equipment, the processing of ingredients, and the chef's operations.

[0065] 2. Using computer vision technology to identify various cooking devices in video streams or image data, and locating and classifying different cooking devices in new video streams or images, including the following steps:

[0066] Firstly, historical kitchen video streams or image data are obtained from the database, and the types and locations of cooking equipment in the video streams or image data are marked to form a sample data set; the sample data set is divided into a training set and a test set; and the training set is used to train a preset convolutional neural network (CNN).

[0067] Use feature detection algorithms such as SIFT and SURF to extract key points and key features in the image, including features such as contours, edges, and textures, so that the convolutional neural network CNN can classify and locate the extracted key features and identify various cooking equipment;

[0068] The performance of the convolutional neural network is tested through the test set, and the convolutional neural network that passes the performance test is deployed into the system.

[0069] Then, according to the preset acquisition frequency, the video stream data or image data of various cooking equipment in the kitchen is acquired in real time to form a real-time kitchen image data set;

[0070] The convolutional neural network analyzes the video stream data frame by frame, uses the region proposal network to generate candidate regions including cooking equipment, and then classifies and regresses the generated candidate regions to determine the category and location of each candidate region; the category and location of each candidate region are labeled;

[0071] Analyze the newly acquired images or video streams frame by frame to identify the type and coordinate information of each cooking device;

[0072] Construct a two-element array of device type and coordinate information ,in, Respectively represent the type and coordinates of the cooking equipment.

[0073] 3. Analyzing the behavior patterns of different cooking devices to analyze the working status of cooking devices includes the following steps:

[0074] After the convolutional neural network identifies the type and coordinate information of each cooking device, it performs behavioral pattern recognition on the identified devices;

[0075] Analyze the behavior patterns of cooking equipment and identify the working status of cooking equipment; specifically:

[0076] Acquiring specific data that can reflect a behavior pattern of the cooking device, wherein the specific data includes thermal image data, sensor data, and audio data of the cooking device;

[0077] By analyzing the thermal image data of the cooking equipment, the heat distribution characteristics of the cooking equipment are identified, and based on the heat distribution characteristics of the cooking equipment, the working status of the cooking equipment is identified; for example, by identifying the thermal image of the gas hood flame, it is determined whether the gas hood is ignited, and whether it is in high, medium or low fire mode.

[0078] By analyzing the sensor data, the pressure distribution characteristics of the cooking equipment are obtained, and the working status of the cooking equipment is identified based on the pressure distribution characteristics of the cooking equipment; for example, the sensor in the pressure cooker measures the pressure in the pot to determine whether the pressure cooker is turned on and the pressure status.

[0079] By analyzing the audio data, the working noise distribution characteristics of the cooking equipment are obtained, and the working status of the cooking equipment is identified based on the working noise distribution characteristics of the cooking equipment.

[0080] 4. Identify the dishes being cooked by the cooking equipment, and after matching the corresponding cooking process with the working status of the cooking equipment, determine whether human supervision is required, including the following steps:

[0081] The method of identifying the dishes being cooked by the cooking device and, in combination with the working state of the cooking device, determining whether personnel supervision is required after matching the corresponding cooking process flow includes:

[0082] Identify the cooking equipment in operation and obtain information about dishes in the cooking equipment;

[0083] Get the working status of the cooking equipment in operation;

[0084] Obtaining a cooking process that matches the dish information from a database, extracting multiple cooking parameters at multiple key nodes from the cooking process, and forming multiple cooking parameter time series; the cooking parameters include cooking equipment type, temperature, pressure, and cooking time;

[0085] According to the relationship between the types of dishes, cooking techniques and equipment types, it is determined whether the dishes need to be supervised during the cooking time, which specifically includes the following steps:

[0086] The method of judging whether a dish needs human supervision during the cooking time according to the association between the dish type, cooking process and equipment type includes:

[0087] Define cooking equipment types and construct cooking equipment type sequences ,in, Indicates cooking equipment;

[0088] Define dish types and construct dish type sequences ,in, Indicates Plant vegetables;

[0089] Define cooking methods and build cooking method sequences ,in, Indicates cooking methods;

[0090] Define the personnel guard rule dictionary, and randomly extract an element from the cooking equipment type sequence, dish type sequence, and cooking method sequence to form a comparison sequence ,in, , , ;

[0091] Compare the comparison sequence with the personnel guard rule dictionary to determine whether the personnel guard rule is met. If so, determine whether the personnel guard rule is met. Types of cooking equipment Types of cooking methods When serving this type of dish, staff supervision is required; otherwise, no staff supervision is required.

[0092] In this embodiment:

[0093] Define the device type: ;

[0094] Define dish type: ;

[0095] Define the cooking method: ;

[0096] Define the personnel guarding rule dictionary: rule = {(oven, cake, electric heating): no personnel guarding is required; (gas hood, cooking, open flame heating): personnel guarding is required; (gas hood, cooking, electric heating): personnel guarding is required; (rice cooker, soup, electric heating): no personnel guarding is required; (gas hood, soup, open flame heating): personnel guarding is required; (pressure cooker, soup, pressurized heating): personnel guarding is required}.

[0097] After inputting the type of cooking equipment, dish type and cooking method, the system outputs the judgment result: whether human supervision is required or not.

[0098] 5. Select a corresponding safety reminder scheme according to the judgment result of the previous step, the safety reminder scheme includes an on-duty reminder scheme and a self-processing reminder scheme, including the following steps:

[0099] Obtain kitchen video stream data and divide the video stream into multiple frame images;

[0100] Analyze multiple frames of images frame by frame to obtain the movement trajectory and behavior pattern of kitchen staff;

[0101] Establish the association between the behavior pattern of kitchen staff and the working status of cooking equipment on the timeline, and calculate the correlation degree 1; specifically:

[0102] Get kitchen staff behavior data from the video stream, including the timestamp and type of each action, such as turning on or off a device.

[0103] Get the working status data of cooking equipment from the video stream, including the state change timestamp and status of each device, for example, on, off.

[0104] Preprocess the behavior data of kitchen staff and the working status data of cooking equipment, and sort the behavior data and equipment status data by time. Use sliding window technology to match behavior and equipment status. If a behavior occurs within a certain time range T before and after the change of equipment status, the two are considered to be related. Count all the related behavior and equipment status pairs, and calculate their ratio a as the correlation degree 1. When the correlation degree 1 is greater than the preset correlation threshold, it is judged that the behavior of the kitchen staff is operating the cooking equipment.

[0105] Establish the association between the behavior patterns of kitchen staff and the dish-making processes, and calculate the second degree of association. Screen the kitchen staff based on the first and second degrees of association to identify the personnel related to dish cooking. Specifically:

[0106] Collect the behavior data of kitchen staff and the processing data of dishes. These data can include: the action time series data of the staff, the timestamp of each action, the specific type of each action (such as cutting vegetables, cooking, plating, etc.), the processing steps of the dishes and their corresponding timestamps.

[0107] Extract useful features from the collected data: action frequency, action duration, action sequence, and the correspondence between actions and dish processing steps.

[0108] Use statistical methods or machine learning algorithms to calculate the degree of association between behavior patterns and dish-making processes. By setting a time window (e.g., 60 seconds), count the frequencies of different actions and steps within each window. Use cosine similarity to calculate the similarity between behavior patterns and dish-making processes.

[0109] Take the calculated cosine similarity b as the second degree of association. If b is greater than the similarity threshold, then it is considered that the action pattern of the personnel is related to dish cooking.

[0110] When personnel guardianship is required, execute the on-duty reminder plan; when personnel guardianship is not required, execute the self-processing reminder plan.

[0111] Among them, when personnel guardianship is required, execute the on-duty reminder plan, including:

[0112] First, we need to obtain real-time video stream data from the kitchen cameras. This can be achieved by using the VideoCapture class in the OpenCV library.

[0113] To identify the personnel related to dish cooking, we can use the deep learning model YOLO to detect humans in the video frames. These models can be trained to identify specific behaviors or clothing in the kitchen, so as to lock the personnel related to dish cooking.

[0114] Optical flow method is a method for estimating the pixel motion in an image sequence. We will use the Lucas-Kanade optical flow method to track the position changes of personnel in consecutive frames. By calculating the optical flow between each pair of consecutive frames, we can obtain the motion vector of each pixel, and then obtain the motion trajectory of the entire target.

[0115] Based on the current and previous motion trajectories, we can use a simple linear prediction model to predict the position of personnel in the next frame. For example, we can use several recent position points to fit a straight line and predict the next point's position.

[0116] If the predicted next position is beyond the preset safety distance from the cooking equipment, the system will issue a warning signal. This can be achieved by sending a warning message or voice to the relevant personnel.

[0117] Among them, when no human guard is required, the self-processing reminder plan is executed, including:

[0118] After acquiring the video stream data, we need to identify the cooking equipment in the video. This can be achieved through deep learning models, such as using object detection models such as YOLO (You Only Look Once) or SSD (Single Shot MultiBoxDetector) to identify and locate cooking equipment in the kitchen.

[0119] Once we have identified the cooking equipment, the next step is to determine the working status of these equipment. This can be done by analyzing the equipment's operating parameters, such as temperature, power consumption, etc., to determine whether the equipment is working properly. If the status of the equipment is abnormal, we will trigger an alert signal.

[0120] When a change in the working state of the device is detected (for example, from on to off), the system will output a reminder signal. This signal can be a sound alarm, SMS notification, or other means to notify relevant personnel.

[0121] The present invention also provides a visual recognition-based unattended cooking reminder system, wherein the system is used to execute the visual recognition-based unattended cooking reminder method.

[0122] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the visual recognition-based unmanned cooking reminder method.

[0123] In the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. The embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above functions defined in the method of the present application are executed. It should be noted that the computer-readable medium mentioned above in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wire segments, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, electrical wire, optical cable, RF, etc., or any suitable combination of the foregoing.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments. Without departing from the principles, the implementation methods of the present invention may be subject to any changes or modifications.

Claims

1. A cooking reminder method based on visual recognition, characterized in that: The method comprises: The video stream or image data in the kitchen is obtained through the camera, and the internal environment parameters of the cooking equipment during cooking are obtained through the sensor; Use computer vision technology to identify various cooking equipment in video streams or image data, obtain kitchen history video streams or image data from a database, mark the types and positions of cooking equipment in the video streams or image data, and form a sample data set; divide the sample data set into a training set and a test set; use the training set to train a preset convolutional neural network, so that the convolutional neural network extracts the contours, edges and texture features of different cooking equipment from the kitchen history video stream data or image data, and learns the features of different cooking equipment; test the performance of the convolutional neural network through the test set, and deploy the convolutional neural network that passes the performance test into the system, obtain video stream data or image data containing various cooking equipment in the kitchen in real time according to the preset acquisition frequency, form a real-time kitchen image data set, and locate and classify different cooking equipment in the new video stream or image; The working state of the cooking equipment is analyzed by analyzing the behavior patterns of different cooking equipment. The heat distribution characteristics of the cooking equipment are identified by analyzing the thermal image data of the cooking equipment. The working state of the cooking equipment is identified based on the heat distribution characteristics of the cooking equipment. The pressure distribution characteristics of the cooking equipment are obtained by analyzing the sensor data. The working state of the cooking equipment is identified based on the pressure distribution characteristics of the cooking equipment. The working noise distribution characteristics of the cooking equipment are obtained by analyzing the audio data. The working state of the cooking equipment is identified based on the working noise distribution characteristics of the cooking equipment. Identify the dishes being cooked by the cooking equipment, and after matching the cooking process with the working status of the cooking equipment, determine whether human supervision is required; A corresponding safety reminder scheme is selected according to the judgment result of the previous step, and the safety reminder scheme includes an on-duty reminder scheme and a self-processing reminder scheme.

2. The method for unattended cooking reminder based on visual recognition according to claim 1, characterized in that: The method of locating and classifying different cooking devices in a new video stream or image includes: The convolutional neural network analyzes the video stream data frame by frame, uses the region proposal network to generate candidate regions including cooking equipment, and then classifies and regresses the generated candidate regions to determine the category and location of each candidate region; Label the category and location of each candidate area; Analyze the newly acquired images or video streams frame by frame to identify the type and coordinate information of each cooking device; Construct a two-element array of device type and coordinate information ,in, Respectively represent the type and coordinates of the cooking equipment.

3. The method for unattended cooking reminder based on visual recognition according to claim 1, characterized in that: The method of identifying the dishes being cooked by the cooking device and, in combination with the working state of the cooking device, determining whether personnel supervision is required after matching the corresponding cooking process flow includes: Identify the cooking equipment in operation and obtain information about dishes in the cooking equipment; Get the working status of the cooking equipment in operation; Obtaining a cooking process that matches the dish information from a database, extracting multiple cooking parameters at multiple key nodes from the cooking process, and forming multiple cooking parameter time series; the cooking parameters include cooking equipment type, temperature, pressure, and cooking time; Based on the relationship between dish types, cooking techniques and equipment types, determine whether the dishes require human supervision during the cooking time.

4. The method for unattended cooking reminder based on visual recognition according to claim 3, characterized in that: The method of judging whether a dish needs human supervision during the cooking time according to the association between the dish type, cooking process and equipment type includes: Define cooking equipment types and construct cooking equipment type sequences ,in, Indicates cooking equipment; Define dish types and construct dish type sequences ,in, Indicates Plant vegetables; Define cooking methods and build cooking method sequences ,in, Indicates cooking methods; Define the personnel guard rule dictionary, and randomly extract an element from the cooking equipment type sequence, dish type sequence, and cooking method sequence to form a comparison sequence ,in, , , ; Compare the comparison sequence with the personnel guard rule dictionary to determine whether the personnel guard rule is met. If so, determine whether the personnel guard rule is met. Types of cooking equipment Types of cooking methods When serving this type of dish, staff supervision is required; otherwise, no staff supervision is required.

5. The method for unattended cooking reminder based on visual recognition according to claim 4, characterized in that: The step of selecting a corresponding safety reminder scheme according to the judgment result of the previous step includes: Obtain kitchen video stream data and divide the video stream into multiple frame images; Analyze multiple frames of images frame by frame to obtain the movement trajectory and behavior pattern of kitchen staff; Establish an association between the behavior pattern of kitchen staff and the working status of cooking equipment on the timeline, and calculate the correlation degree one. Obtain the behavior data of kitchen staff from the video stream, including the timestamp and operation type of each operation. Obtain the working status data of cooking equipment from the video stream, including the state change timestamp and state of each device. Preprocess the behavior data of kitchen staff and the working status data of cooking equipment, sort the behavior data and the equipment status data by time, and use the sliding window technology to match the behavior and the equipment status. If a certain behavior occurs within a certain time range T before and after the equipment status change, the two are considered to be associated. Count all the associated behavior and equipment status pairs, and calculate their ratio a as the correlation degree one. Establish the association between the behavior pattern of kitchen staff and the dish process, and calculate the second degree of association. Collect the behavior data of kitchen staff and the processing data of dishes. Extract useful features from the collected data: action frequency, action duration, action sequence, and the correspondence between actions and dish processing steps. Use statistical methods or machine learning algorithms to calculate the association between the behavior pattern and the dish process. By setting a time window, count the frequencies of different actions and steps in each window, use cosine similarity to calculate the similarity between the behavior pattern and the dish process, and use the calculated cosine similarity b as the second degree of association. Screen the kitchen staff according to the first and second degrees of association to identify the staff related to dish cooking. When human guarding is required, the on-duty reminder plan is executed; when human guarding is not required, the self-processing reminder plan is executed.

6. The method for unattended cooking reminder based on visual recognition according to claim 5, characterized in that: When personnel guarding is required, the guard reminder plan is executed, including: Obtain kitchen video stream data, identify and lock people related to cooking from the video stream data; The optical flow method is used to analyze multiple consecutive frame images containing personnel information related to the cooking of dishes frame by frame to obtain the movement trajectory of the personnel related to the cooking of dishes; Predict the next location of the staff related to the dish based on the acquired action trajectory. If the next location is more than a preset safety distance from the dish cooking equipment, send a reminder signal 1 to notify the staff related to the dish to return to the working range. When no personnel guarding is required, the self-processing reminder scheme is executed, including: Obtain kitchen video stream data, and identify cooking equipment related to cooking dishes from the video stream data; Identify the working status of corresponding cooking equipment; When the working status changes, a second reminder signal is output to notify the staff related to the cooking of the dish to determine whether the working status is normal.

7. A cooking unattended reminder system based on visual recognition, characterized in that: The system is used to execute the visual recognition-based unattended cooking reminder method described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the visual recognition-based unattended cooking reminder method described in any one of claims 1 to 6.

Images