Information generation method and system based on kitchen equipment
By detecting kitchen equipment operation information and the pictures taken by the camera, generating a timeline and combining operation events, the problem of incomplete kitchen equipment operation records in the existing technology is solved, and more detailed and useful graphic and text process templates are provided, which improves the User cooking experience.
Patent Information
- Application Number
- CN202410624114.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-19
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-05-19
AI Technical Summary
In the prior art, kitchen equipment operation recording is performed only through an image scheme, which makes it difficult to determine the operation of single image recognition, and the recording effect is relatively primitive, making it difficult for users to obtain valuable cooking experience directly from the output device.
通过检测设备操作信息和摄像头拍摄的画面,生成时间轴,去重后合并同类操作事件,生成流程模板并添加配图,供用户填写和分享。
Improve the integrity and usability of records, users can gain cooking experience more intuitively through the generated graphic and text process templates, and solve the problem of incomplete content.
Smart Images

Figure CN118540535B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to image processing and content generation technology, and in particular to an information generation method and system based on kitchen equipment. Background Art
[0002] In the prior art, there are schemes to record the user's cooking process, thereby generating some records related to the cooking process. The prior art uses a camera to shoot the user's operation to prompt the user to continue the operation. However, the relevant scheme currently only uses image schemes for recording, and image recognition may be difficult to determine for some operations. In addition, the current records are still in a relatively primitive stage, and it is usually difficult for users to directly use the content recorded by the output device as experience. Summary of the invention
[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides a method and system for generating information based on kitchen equipment, which can improve the shortcomings of the single image recognition solution and the problem of poor effect of the original recording solution.
[0004] On the one hand, an embodiment of the present application provides a method for generating information based on kitchen equipment, comprising the following steps:
[0005] Detecting operation information of a device, and recording a first operation event and a corresponding first operation time according to the operation information of the device, wherein the operation information is information of a user directly operating the device through a device panel;
[0006] Identify a second operation event of the user and a corresponding second operation time through the picture captured by the camera, and capture at least a portion of the picture during the period when the second operation event occurs, where the picture is associated with the second operation event;
[0007] generating a timeline according to the existing sequence of the first operation event and the second operation event;
[0008] De-duplicate the events on the timeline, and obtain a revised timeline based on the de-duplicate results;
[0009] Generate a process template according to the modified timeline and the screens associated with each second operation event, wherein the process template includes an input container for a user to fill in content;
[0010] When the user completes the input, the process template is typeset and a shareable link is generated.
[0011] In some embodiments, deduplicating events on the timeline and obtaining a revised timeline according to the deduplicated results includes:
[0012] Classifying the same type of operation events to modify the time axis, the operation events including the first operation event or the second operation event;
[0013] Wherein, according to the time interval of similar operation events on the time axis, two adjacent operation events with a time interval less than a threshold are determined as operation events to be merged, and when there is no preset operation event that separates the operation events to be merged between the two operation events to be merged, the two operation events are merged;
[0014] Different preset operation events are set for different operation events; the position of the merged operation event on the time axis is determined according to whether there are other operation events between the two operation events before the merger, and the types of the other operation events;
[0015] When there is no other operation between two merged operation events, the position of the merged operation event on the timeline is determined based on the average value of the original positions of the two operation events on the timeline; when there are other operation events between the two merged operation events, the position of the merged operation event on the timeline is determined based on the types of the two merged operation events and the types of other intermediate operation events.
[0016] In some embodiments, the identification of the second event specifically includes:
[0017] Determine the recognition range according to the location of the target object, and determine the occurrence of the event according to the elements recognized in the recognition range;
[0018] wherein, the state of the first object is identified, and when the state of the first object changes, the current second operation event is determined according to the state change; the first object includes a pot cover;
[0019] When the first object is in the first state, other second operation events are recognized; when the first object is in the second state, recognition of other second operation events is stopped, wherein the other second operation events include feeding, stir-frying or taking out ingredients.
[0020] In some embodiments, the recognition range is determined according to the position of the target object, specifically:
[0021] Identify the location of the kitchenware, and determine a first range as the identification range according to the location of the kitchenware and the size of the kitchenware in the image;
[0022] When identifying the other second operation event, the second range is used as the identification range, the second range includes the first range, and the second range is larger than the first range.
[0023] In some embodiments, when the state of the first object is identified, it is performed by extracting picture frames at time intervals, and when the other second operation events are identified, it is performed by using video clips.
[0024] In some embodiments, the method of triggering the second operation event recognition is:
[0025] The edge computing power of the device where the camera is located is used to detect the amplitude of video changes, and mark the video clips with a change amplitude greater than a threshold. The server identifies the video clips before and after the marked location.
[0026] In some embodiments, the device includes a cooking stove, and the first operation event includes a user's selection of a stove mode, adjustment of a cooking power of the stove, and shutting down and starting up the stove.
[0027] In some embodiments, the following steps are also included:
[0028] After capturing at least part of the screen during the occurrence of the second operation event, a graphic is generated in the corresponding second operation event according to the process template, and the graphic is a static image or a dynamic image, wherein the static image selects a picture that meets the clarity requirement from the part of the screen.
[0029] Some embodiments further include the following step: selecting words used in generating content according to the length of time intervals between each first operation event and / or second operation event.
[0030] An information generation system based on kitchen equipment, comprising:
[0031] Memory, used to store programs;
[0032] A processor is used to load the program to implement the information generation method based on kitchen equipment.
[0033] Through the embodiment of the present application, the operation information of the device is detected, and a first operation event and a corresponding first operation time are recorded according to the operation information of the device, wherein the operation information is information of the user directly operating the device through the device panel; a second operation event of the user and a corresponding second operation time are identified through the picture taken by the camera, and at least part of the picture during the period when the second operation event occurs is captured, and the picture is associated with the second operation event; it can be seen that the present solution will simultaneously record the user's operation on the device, such as the operation of the gas stove, etc., and will also analyze the user's cooking process video taken by the device such as the range hood; it solves the problem that the existing technology currently only records information through the picture, resulting in incomplete generated content; then, the present application Please implement the method of generating a timeline according to the existing order of the first operation event and the second operation event; deduplicate each event on the timeline and obtain a revised timeline according to the result of deduplication; generate a process template according to the revised timeline and the screen associated with each second operation event, wherein the process template includes an input container for the user to fill in the content; when the user completes the input, the process template is typeset and a shareable link is generated. It can be seen that this scheme will sort the two time categories recorded by the system, deduplicate and simplify them, and generate a graphic process template for the user to edit, and then generate content that can be shared with other users according to the complete content filled in by the user, thereby solving the problem of limited availability of content generated by the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments.
[0035] Figure 1 It is a flow chart provided by an embodiment of the present application;
[0036] Figure 2 It is a schematic diagram of a time axis provided in an embodiment of the present application;
[0037] Figure 3 It is a system block diagram provided in an embodiment of the present application. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described through the implementation method with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.
[0039] Reference Figure 1 and Figure 2, the embodiment of the present application discloses a method for generating information based on kitchen equipment, an exemplary scenario, such as Figure 3 As shown, it involves a stove (with an operation panel 1) and a range hood (with an operation panel 2 and a camera), the two communicate with each other, and the range hood communicates with the server. In this embodiment, it involves recording the user's operation on the functions on the operation panels of the two devices, that is, the first operation event in this embodiment, and recording the user's operation behavior obtained by shooting with the camera set by the range hood, that is, the second operation event in this embodiment; in some embodiments, the device can refer to a cooking stove, and the first operation event includes the user's selection of the stove mode, adjusting the cooking fire size of the stove, and shutting down and starting the stove. It can be understood that the user can start these services, and the corresponding device will record the operation event when it is started and upload it to the server. After processing, the server returns it to the user terminal, and the user terminal can further edit it, and then generate content that can be shared to share with other users.
[0040] The method comprises the following steps:
[0041] S1. Detect operation information of a device, and record a first operation event and a corresponding first operation time according to the operation information of the device, wherein the operation information is information of a user directly operating the device through a device panel.
[0042] In this step, the user's operation of several devices through the operation panel is recorded. For example, in some solutions, the stove and the range hood form a system, which communicate with each other and are connected to the server. The above-mentioned device can independently record the operation time and operation content of the user operating the device through the panel, that is, the first operation event referred to in this step. In some embodiments, the device can refer to the stove alone. The stove includes a gas stove, an induction cooker, etc. In this solution, the device is connected to the network and uploads the operation information of this cooking.
[0043] S2. Identify a second operation event of the user and a corresponding second operation time through the picture taken by the camera, and capture at least a portion of the picture during the period when the second operation event occurs, where the picture is associated with the second operation event.
[0044] In this step, the second operation event is mainly the operation performed by the user around the kitchenware during the cooking process, including but not limited to opening / closing the pot lid, putting ingredients, taking ingredients, stir-frying, stirring, etc. In this embodiment, the cooking process can be filmed by a camera provided on the range hood associated with the cooker.
[0045] It is understandable that operations such as putting in ingredients, taking out ingredients, stir-frying, and stirring need to be performed with the lid open. Therefore, before identifying these operations, you can first determine whether these actions need to be identified by the state of the pot lid. Since these actions are usually continuous actions and need to be identified through video clips, the model requires high computing power and identification is also time-consuming. Therefore, determining whether other actions need to be further identified by the state of the pot lid can effectively reduce the amount of calculation and optimize the overall processing time.
[0046] Therefore, in some embodiments, the identification of the second event specifically includes:
[0047] The recognition range is determined according to the position of the target object, and the occurrence of the event is determined according to the elements recognized in the recognition range. It can be understood that when identifying the state of the pot cover, it is actually only necessary to identify whether the position where the kitchenware is located is covered with a pot cover. Switching between the states of whether there is a pot cover can determine the two events of opening the pot cover and covering the pot cover. At the same time, it can also avoid introducing more useless recognition information and reducing the recognition accuracy. In this solution, the size image of the model input size can be intercepted based on the position of the kitchenware as the center for recognition. Therefore, the state of the first object is identified, and when the state of the first object changes, the current second operation event is determined according to the state change; the first object includes the pot cover.
[0048] When the first object is in the first state, other second operation events are recognized; when the first object is in the second state, recognition of other second operation events is stopped, wherein the other second operation events include feeding, stir-frying or taking out ingredients.
[0049] In some embodiments, different recognition size ranges can be used to identify the state of a pot lid and other second operation events. Only at least one frame of image is required to identify the state of a pot lid (it is only necessary to identify whether there is a pot lid on the kitchenware). Therefore, the image range only needs to include the range where the kitchenware is located during recognition. When recognizing actions such as stir-frying, it is a continuous action, and multiple image frames are required to distinguish. Therefore, when recognizing the state of the first object, the image frames are extracted at time intervals, and when recognizing the other second operation events, video clips are used. Although both types of recognition occur around kitchenware, the information required to recognize the state of the pot lid is different from that required for specific dynamic actions. Therefore, the recognition range is determined according to the position of the target object, specifically:
[0050] Identify the location of the kitchenware, and determine a first range as the identification range according to the location of the kitchenware and the size of the kitchenware in the image;
[0051] Among them, when identifying the other second operation events, the second range is used as the identification range, the second range includes the first range, and the second range is larger than the first range. It can be understood that for identifying some specific operation events, such as more complex actions such as feeding, taking materials and stir-frying, providing information for other second operation events through a larger range can improve the accuracy of identification.
[0052] In some embodiments, the method of triggering the second operation event recognition is:
[0053] The edge computing power of the device where the camera is located is used to detect the amplitude of the video change, and the video clips with a change amplitude greater than the threshold are marked. The server identifies the video clips before and after the marked position. It can be understood that the recognition process can be optimized by using the device to perform edge recognition, identify the amplitude of the video change, and then hand it over to the server for specific recognition of specific complex actions. On the one hand, the edge computing power of the device is used to make relatively simple marks. The device can simplify the video to be transmitted according to the results of the marking. For example, some scenes that have not undergone substantial changes (that is, the changes do not continue to exceed the preset amplitude) are discarded to reduce transmission. On the server side, the deployed model can be upgraded at any time to recognize more and more complex actions.
[0054] S3: Generate a timeline according to the existing sequence of the first operation event and the second operation event.
[0055] It is understandable that each first operation event and each second operation event can be mapped to the event axis in sequence. Figure 2 As shown in the figure, the timeline is finally mapped to the generated content template. The events on the timeline will generate content according to certain rules and mapping relationships based on the time between the previous event and the content of the event itself.
[0056] S4. De-duplicate the events on the timeline, and obtain a revised timeline based on the de-duplicate results. It is understandable that during the recognition process, due to problems with the recognition model or the user's cooking habits, actions that can be performed coherently are separated, but these unexpected recognition results are dispensable for the final generated content, and more streamlined content is more conducive to improving the quality of the content. Therefore, events that meet some conditions can be deleted.
[0057] Specifically, in some embodiments, deduplication of events on the timeline and obtaining a revised timeline according to the deduplication result include:
[0058] The timeline is modified by classifying similar operation events, wherein the operation events include the first operation event or the second operation event. It is understandable that, whether for the first operation event or the second operation event, when deduplication is required, the operation events need to be classified first.
[0059] Among them, according to the time interval of similar operation events on the time axis, two adjacent operation events with a time interval less than a threshold are determined as operation events to be merged, and when there is no preset operation event that separates the operation events to be merged between the two operation events to be merged, the two operation events are merged. Usually, the basis for merging is that the time interval between the two events is relatively small, and at the same time, it is required that nothing that necessarily interrupts the two events occurs between the two times. For example, if the firepower of the stove is adjusted between two stir-fries, stir-frying is actually a continuous action, and adjusting the firepower is just the user's adjustment, which is actually irrelevant to the order of the two stir-fries. At this time, if the event interval between the two stir-fries is short, they can be merged. Assuming that there is stir-frying between the two feedings, in this example, the two feedings are in a sequential order, so they will not be merged. Different preset operation events are set for different operation events, that is, the events that separate different operation events to be merged are not the same; the position of the merged operation event on the time axis is determined according to whether there are other operation events between the two operation events before the merger, and the types of other operation events that exist. It can be understood that the two merged operation events need to be remapped to the time axis.
[0060] In this embodiment, when there is no other operation between the two merged operation events, the position of the merged operation event on the time axis is determined according to the average value of the original positions of the two operation events on the time axis; when there is other operation events between the two merged operation events, the merged operation event is determined according to the types of the two merged operation events and the types of other intermediate operation events, using the earlier or later position of the two merged operation events as the position of the merged operation event on the time axis. It can be understood that in this embodiment, the position of the merged operation event is determined based on different situations.
[0061] S5. Generate a process template according to the corrected timeline and the screens associated with each second operation event, wherein the process template includes an input container for the user to fill in the content. It is understandable that according to each event occurring on the timeline, the occurrence interval of each event can be recorded, and for the second operation event, a related screen can be configured. Then generate a process template, which provides a place for users to edit graphics and texts, so that users can more conveniently modify the graphics and text content, and the system can help users generate graphics and texts. For example, according to the timeline, event 1 is to turn on the fire to high heat, the time is 00.00; event 2 is to add materials, the time is 00.20; time 3 is stir-frying, the time is 00.30; then the generated content can be: turn on the fire to medium heat, add [materials to be filled], and stir-fry after heating for 20 seconds. At this time, the user can edit the materials to be added as peanut oil and minced garlic. Correspondingly, the acquired screen can be added before and after the corresponding text.
[0062] It is understandable that when generating a template, the description of the operation event itself and the time description between two operations are mainly targeted. Different description templates can be configured for different operations and different operation intervals.
[0063] Some embodiments further include the following step: selecting words used in generating content according to the length of time intervals between each first operation event and / or second operation event.
[0064] For example, different adjectives can be selected according to different time lengths. For example, if the time between covering the pot and opening the pot is relatively long, it can be described as stewing, and if the time is relatively short, it can be described as baking, etc. These embodiments help optimize the description words and further optimize the quality of content generation.
[0065] In some embodiments, the following steps are also included:
[0066] After capturing at least part of the screen during the occurrence of the second operation event, a picture is generated in the corresponding second operation event according to the process template, and the picture is a static picture or a dynamic picture, wherein the static picture selects a picture that meets the clarity condition from the partial screen. It is understandable that in the template, the corresponding second operation event can be accompanied by the captured image. This can optimize the generated content. Among them, it can be determined whether to configure a static picture or a dynamic picture according to the user's preferences or for different types of operations. The user can also delete or replace unsatisfactory pictures.
[0067] S6. When the user completes the input, the process template is typeset and a shareable link is generated. It is understandable that the user can fill in and modify the content that he wants to add or change based on the content template automatically generated by the system, and then request the system to typeset it. After typeset, the link can be shared with friends like an article.
[0068] Through the embodiment of the present application, the operation information of the device is detected, and a first operation event and a corresponding first operation time are recorded according to the operation information of the device, wherein the operation information is information of the user directly operating the device through the device panel; a second operation event of the user and a corresponding second operation time are identified through the picture taken by the camera, and at least part of the picture during the period when the second operation event occurs is captured, and the picture is associated with the second operation event; it can be seen that the present solution will simultaneously record the user's operation on the device, such as the operation of the gas stove, etc., and will also analyze the user's cooking process video taken by the device such as the range hood; it solves the problem that the existing technology currently only records information through the picture, resulting in incomplete generated content; then, the present application Please implement the method of generating a timeline according to the existing order of the first operation event and the second operation event; deduplicate each event on the timeline and obtain a revised timeline according to the result of deduplication; generate a process template according to the revised timeline and the screen associated with each second operation event, wherein the process template includes an input container for the user to fill in the content; when the user completes the input, the process template is typeset and a shareable link is generated. It can be seen that this scheme will sort the two time categories recorded by the system, deduplicate and simplify them, and generate a graphic process template for the user to edit, and then generate content that can be shared with other users according to the complete content filled in by the user, thereby solving the problem of limited availability of content generated by the system.
[0069] The embodiment of the present application discloses an information generation system based on kitchen equipment, including:
[0070] Memory, used to store programs;
[0071] A processor is used to load the program to implement the information generation method based on kitchen equipment.
[0072] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0073] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.
Claims
1. A method for generating information based on kitchen equipment, characterized in that: The following steps are involved: Detecting operation information of a device, and recording a first operation event and a corresponding first operation time according to the operation information of the device, wherein the operation information is information of a user directly operating the device through a device panel; Identify a second operation event of the user and a corresponding second operation time through the picture captured by the camera, and capture at least a portion of the picture during the period when the second operation event occurs, where the picture is associated with the second operation event; generating a timeline according to the existing sequence of the first operation event and the second operation event; De-duplicate the events on the timeline, and obtain a revised timeline based on the de-duplicate results; Generate a process template according to the modified timeline and the screens associated with each second operation event, wherein the process template includes an input container for a user to fill in content; When the user completes the input, the process template is typeset and a shareable link is generated; Deduplication of events on the timeline is performed, and a revised timeline is obtained according to the deduplication result, including: Classifying the same type of operation events to modify the time axis, the operation events including the first operation event or the second operation event; Wherein, according to the time interval of similar operation events on the time axis, two adjacent operation events with a time interval less than a threshold are determined as operation events to be merged, and when there is no preset operation event that separates the operation events to be merged between the two operation events to be merged, the two operation events are merged; Different preset operation events are set for different operation events; the position of the merged operation event on the time axis is determined according to whether there are other operation events between the two operation events before the merger, and the types of the other operation events; When there is no other operation between two merged operation events, the position of the merged operation event on the timeline is determined based on the average value of the original positions of the two operation events on the timeline; when there are other operation events between the two merged operation events, the position of the merged operation event on the timeline is determined based on the types of the two merged operation events and the types of other intermediate operation events.
2. The method for generating information based on kitchen equipment according to claim 1, characterized in that: The identification of the second event specifically includes: Determine the recognition range according to the location of the target object, and determine the occurrence of the event according to the elements recognized in the recognition range; wherein, the state of the first object is identified, and when the state of the first object changes, the current second operation event is determined according to the state change; the first object includes a pot cover; When the first object is in the first state, other second operation events are recognized; when the first object is in the second state, recognition of other second operation events is stopped, wherein the other second operation events include feeding, stir-frying or taking out ingredients.
3. The method for generating information based on kitchen equipment according to claim 2, characterized in that: Determine the recognition range based on the location of the target object, specifically: Identify the location of the kitchenware, and determine a first range as the identification range according to the location of the kitchenware and the size of the kitchenware in the image; When identifying the other second operation event, the second range is used as the identification range, the second range includes the first range, and the second range is larger than the first range.
4. The method for generating information based on kitchen equipment according to claim 3, characterized in that: When identifying the state of the first object, it is performed by extracting picture frames at time intervals, and when identifying the other second operation events, it is performed by using video clips.
5. The method for generating information based on kitchen equipment according to claim 4, characterized in that: The method of triggering the second operation event recognition is: The edge computing power of the device where the camera is located is used to detect the amplitude of video changes, and mark the video clips with a change amplitude greater than a threshold. The server identifies the video clips before and after the marked location.
6. The method for generating information based on kitchen equipment according to claim 1, characterized in that: The device includes a cooking stove, and the first operation event includes a user's selection of a stove mode, adjustment of the cooking power of the stove, and operations of shutting down and starting the stove.
7. The method for generating information based on kitchen equipment according to claim 1, characterized in that: The following steps are also included: After capturing at least part of the screen during the occurrence of the second operation event, a graphic is generated in the corresponding second operation event according to the process template, and the graphic is a static image or a dynamic image, wherein the static image selects a picture that meets the clarity requirement from the part of the screen.
8. The method for generating information based on kitchen equipment according to claim 1, characterized in that: The following steps are also included: The words used when generating the content are selected according to the length of the time interval between each first operation event and / or second operation event.
9. An information generation system based on kitchen equipment, characterized in that: include: Memory, used to store programs; A processor, used to load the program to implement the information generation method based on kitchen equipment as described in any one of claims 1-8.
Citation Information
Patent Citations
Steaming and baking device and method with cooking sharing function
CN108742125A
Menu file display method and device, storage medium and electronic device
CN111104194A
Automatic generation method and device of cloud menu and storage medium
CN117056560A
Touch screen type control panel of gas cooker
CN218626952U
Information processing system, information processing device, information processing method, and program
JP2019079483A
