Method and system for intelligent analysis and interaction of live-action video in catering service scene, electronic device and medium

Abstract

Embodiments of the present application provide a kind of real scene video intelligent analysis and interaction method, system, electronic equipment and medium in catering service scene.The method comprises: based on the at least one monitoring video of the restaurant to be monitored, key video frames are obtained from the monitoring video, at least one video segment to be processed is obtained;For any the video segment to be processed, based on the preset video analysis model, the label information corresponding to the video segment to be processed is obtained;The label information at least includes dish name and production link;In response to the query request of user, according to the label information corresponding to the monitoring video or each the video segment to be processed, the display video corresponding to the query request is obtained, and the display video is output.It improves video query efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, system, electronic device, and medium for intelligent analysis and interaction of real-scene video in a catering service scenario. Background Technology

[0002] Currently, most public places, including restaurants, have adopted surveillance systems to ensure venue safety and facilitate post-incident review. Restaurant managers can monitor personnel and restaurant workflows by reviewing surveillance videos, and customers can also use the videos to fulfill personalized query requests.

[0003] However, current surveillance videos are often long-term continuous data streams, which makes it inefficient for users to query surveillance videos. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, electronic device, and medium for intelligent analysis and interaction of real-scene videos in a catering service scenario, in order to solve the problem of how to improve video query efficiency. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, a method for intelligent analysis and interaction of real-scene video in a catering service scenario is provided, the method comprising:

[0006] Based on at least one surveillance video of the restaurant to be monitored, key video frames are obtained from the surveillance video to obtain at least one video segment to be processed.

[0007] For any of the video segments to be processed, based on a preset video analysis model, the tag information corresponding to the video segment to be processed is obtained; the tag information includes at least the dish name and the preparation process;

[0008] In response to a user's query request, based on the tag information corresponding to the monitoring video or each of the video segments to be processed, the display video corresponding to the query request is obtained and the display video is output.

[0009] In a second aspect of this invention, a real-scene video intelligent analysis and interaction system for catering service scenarios is also provided, the system comprising:

[0010] The first acquisition module is used to acquire key video frames from at least one monitoring video of the restaurant to be monitored, and obtain at least one video segment to be processed.

[0011] The second acquisition module is used to acquire the tag information corresponding to any of the video segments to be processed based on a preset video analysis model; the tag information includes at least the dish name and the preparation process.

[0012] The third acquisition module is used to respond to the user's query request, acquire the display video corresponding to the query request based on the tag information corresponding to the monitoring video or each of the video segments to be processed, and output the display video.

[0013] In a third aspect of the present invention, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0014] Memory, used to store computer programs;

[0015] When a processor executes a program stored in memory, it implements the method described in the first aspect above.

[0016] In a fourth aspect of the invention, a computer-readable storage medium is also provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0017] In a fifth aspect of the invention, a computer program product comprising instructions is also provided, which, when run on a computer, causes the computer to perform the method described in the first aspect.

[0018] The present invention provides a method for intelligent analysis and interaction of real-scene video in a catering service scenario. This method obtains at least one video segment to be processed by acquiring key video frames from at least one monitoring video of a restaurant to be monitored. For any one of the video segments to be processed, based on a preset video analysis model, it acquires corresponding tag information. The tag information includes at least the dish name and preparation stage. In response to a user's query request, based on the monitoring video or the tag information corresponding to each of the video segments to be processed, it acquires and outputs the display video corresponding to the query request. Thus, by acquiring key video frames from the monitoring video of the restaurant to be monitored, the present invention can retain the most informative video segments to be processed. Simultaneously, by acquiring tag information such as dish names and preparation stages from the video segments to be processed through a preset video analysis model, it can improve video analysis efficiency. Furthermore, when a user triggers a query request, the corresponding display video can be obtained and output to the user based on the monitoring video and tag information. This can meet various user query needs while improving video query efficiency and, to a certain extent, ensuring the validity of the output display video. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0020] Figure 1 This is a flowchart illustrating the steps of a real-scene video intelligent analysis and interaction method in a catering service scenario according to an embodiment of the present invention.

[0021] Figure 2 This is a flowchart illustrating a real-scene video intelligent analysis and interaction method in a catering service scenario according to an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of label information in an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of the overall processing flow of the restaurant video intelligent analysis system in an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of the overall processing flow of the restaurant video intelligent analysis system in an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of the structure of a real-scene video intelligent analysis and interaction system in a catering service scenario according to an embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In embodiments of this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0029] Figure 1 This is a flowchart illustrating the steps of a real-scene video intelligent analysis and interaction method in a catering service scenario according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0030] Step 101: Based on at least one surveillance video of the restaurant to be monitored, obtain key video frames from the surveillance video to obtain at least one video segment to be processed.

[0031] Step 102: For any of the video segments to be processed, obtain the tag information corresponding to the video segment to be processed based on a preset video analysis model; the tag information includes at least the dish name and the preparation process.

[0032] Step 103: In response to the user's query request, obtain the display video corresponding to the query request based on the tag information corresponding to the monitoring video or each of the video segments to be processed, and output the display video.

[0033] Regarding steps 101-103 above, this embodiment of the invention can be applied to any video surveillance system, for example, it can be the surveillance system of any restaurant; this embodiment of the invention does not limit this. The restaurant to be monitored can be any restaurant with a need to query surveillance video, such as a Chinese restaurant, a Western restaurant, etc.; this embodiment of the invention does not limit the type of restaurant. Furthermore, the aforementioned restaurant to be monitored can be a restaurant equipped with a video surveillance system, which can have surveillance cameras located in different areas of the restaurant. These surveillance cameras are used to collect surveillance video from different locations within the restaurant. Specifically, the surveillance cameras installed in the restaurant to be monitored can be located in the dining area, the food delivery area, and the kitchen, etc., correspondingly, comprehensive monitoring of all areas of the restaurant to be monitored can be achieved.

[0034] The aforementioned surveillance video refers to the video stream captured by the surveillance cameras in the restaurant to be monitored. One surveillance video can correspond to the video stream captured by one surveillance camera during a certain monitoring time period. Accordingly, different surveillance videos can correspond to surveillance cameras in different areas and / or monitoring time periods. Specifically, the video streams captured by each surveillance camera in the restaurant to be monitored can be stored in a preset video library, so that the embodiments of the present invention can obtain and process the surveillance video of the restaurant to be monitored from the preset video library.

[0035] Furthermore, embodiments of the present invention can process every single surveillance video, or they can select a portion of the surveillance videos for processing. Optionally, the intelligent analysis and interactive operation of real-scene videos in the catering service scenario of the present invention can be performed every time a new surveillance video is added to the preset video library. Of course, the intelligent analysis and interactive operation of real-scene videos in the catering service scenario can also be performed periodically on the surveillance videos in the preset video library according to a preset processing cycle. This can be set according to actual needs, and the embodiments of the present invention do not impose any restrictions on this.

[0036] The aforementioned key video frames can be representative video frame images, video frame images containing a lot of important information, or video frame images representing changes in video content. Specifically, a surveillance video can contain multiple key video frames, and correspondingly, the aforementioned video segment to be processed can be obtained by stitching together multiple key video frames contained in a surveillance video.

[0037] Specifically, step 101 above can be implemented using a pre-trained video information extraction model. This model can analyze video frames and extract representative video frames as key video frames. Specifically, the training process of the video information extraction model can be as follows: multiple sample videos are input into an initial model. The initial model identifies the key frames based on the video content of each sample video and outputs the identified key frames. Simultaneously, the sample videos can be manually labeled beforehand to mark reference key frames. Then, by comparing the sample key frames with the reference key frames, the model parameters of the initial model can be adjusted. The process of inputting multiple sample videos into the initial model is repeated until the training time reaches a preset duration or the model's loss function value reaches a preset value. At this point, the current initial model is considered the successfully trained video information extraction model.

[0038] The video analysis model mentioned above can also be pre-trained. It is used for deep understanding and multi-dimensional analysis of video clips, extracting tag information from the video clips. This tag information can include at least the dish name and the cooking stage. The dish name can be the name of the dish appearing in the video clip, such as braised pork or sweet and sour pork. The cooking stage can be any stage in the cooking process of the dish appearing in the video clip. Specifically, the cooking stage can be any of the following: obtaining the ingredients, preparing the ingredients, cooking, or serving.

[0039] Specifically, the training process of the aforementioned video analysis model can be as follows: Multiple sample videos, each corresponding to different dishes and preparation stages, are pre-collected and input into the initial analysis model. The initial analysis model identifies the label information of the keyframe images in the sample videos and outputs the identified dish names and preparation stages as sample information. Simultaneously, the sample videos can be manually labeled beforehand, with the corresponding label information serving as reference information. The model parameters of the initial analysis model can then be adjusted by comparing the sample information with the reference information. The process of inputting multiple sample videos into the initial analysis model is repeated until the training time reaches a preset duration or the model's loss function value reaches a preset value. At this point, the current initial analysis model is considered the successfully trained video analysis model.

[0040] The aforementioned query request can be triggered by user interaction, allowing users to perform different interactive operations based on their actual query needs. Accordingly, the user's query intent can be determined based on their interaction, leading to the generation of a query request, which may include environmental queries, menu queries, etc. Furthermore, in this embodiment of the invention, the display video corresponding to the query request can be obtained and output based on the tag information corresponding to the monitoring video or each video segment to be processed.

[0041] Specifically, when the query request is an environmental query, the surveillance video can be directly output as the display video. This can be done by selecting a portion of the surveillance video corresponding to different restaurant areas from a preset video library, or by selecting surveillance videos corresponding to different monitoring time periods from the preset video library. The settings can be configured according to actual needs, and this embodiment of the invention does not impose any limitations on this. Optionally, the above query request can be a query request targeting a specific restaurant area or a specific monitoring time period. Accordingly, this embodiment of the invention can retrieve relevant real-time or historical surveillance videos from the preset video library according to the target restaurant area or target monitoring time period indicated in the query request, facilitating user browsing.

[0042] Accordingly, when the query request is for a dish, the video segment that matches the target dish indicated in the query request can be selected as the display video based on the tag information corresponding to each video segment to be processed. This embodiment of the invention does not impose any limitations on this.

[0043] Optionally, before obtaining key video frames from the surveillance video, the embodiments of the present invention may further include:

[0044] S1011. Compress the surveillance video.

[0045] Specifically, intelligent compression algorithms can be used to compress surveillance videos. For example, intra-frame compression, inter-frame compression, and other algorithms can be used to process surveillance videos. The intelligent compression algorithm can be set according to actual needs, and this embodiment of the invention does not impose any limitations on it.

[0046] This reduces data redundancy, improves video processing efficiency, and further enhances subsequent video query efficiency.

[0047] Optionally, the operation of outputting the display video described above may specifically include, in embodiments of the present invention:

[0048] S1031. Based on the display video, generate summary text for the display video.

[0049] S1032. Output the summary text and the display video.

[0050] The aforementioned summary text can be a brief introduction to the displayed video, generated based on the tag information of the video segments to be processed contained in the displayed video and the video content, or it can be a pre-written text description for different query requests, which can be a text description with a preset number of words. This embodiment of the invention does not limit this.

[0051] Specifically, before outputting the above demonstration video, a preset watermark can be added to the demonstration video. The preset watermark can be a light-colored string, which can ensure the video copyright while avoiding affecting the video effect.

[0052] Furthermore, when outputting the demonstration video, a download link can be provided for users to download, or the video can be directly output to a preset player for users to view. The output method of the demonstration video can be set according to actual needs, and this embodiment of the invention does not impose any restrictions on it.

[0053] This allows for multi-dimensional video presentations, enhancing the diversity of video displays.

[0054] Optionally, embodiments of the present invention may further include:

[0055] S1021. Receive the user's query command.

[0056] S1022. Perform semantic parsing on the query instruction to obtain a query request that matches the user's query intent.

[0057] The query command can be a text command or a voice command, and this embodiment of the invention does not impose any limitations on this. Accordingly, upon receiving a user's query command, it indicates that the user has a video query requirement. At this time, the query command can be semantically parsed to obtain a query request that matches the user's query intent.

[0058] Specifically, natural language processing (NLP) techniques can be used to semantically analyze query commands to determine the user's query intent. After semantic analysis using NLP, query vectors can be constructed to obtain query requests that match the user's intent. Furthermore, step 103 above can utilize large-scale model semantic matching and feature vector retrieval algorithms to obtain the display video corresponding to the query request.

[0059] In summary, the intelligent analysis and interaction method for real-time video in a catering service scenario provided by this invention obtains at least one video segment to be processed by acquiring key video frames from at least one monitoring video of the restaurant to be monitored. For any one of the video segments to be processed, the method obtains the corresponding tag information based on a preset video analysis model. The tag information includes at least the dish name and the preparation process. In response to a user's query request, the method obtains the display video corresponding to the query request based on the monitoring video or the tag information corresponding to each of the video segments to be processed, and outputs the display video. Thus, by acquiring key video frames from the monitoring video of the restaurant to be monitored, this invention can retain the most informative video segments to be processed. Simultaneously, by obtaining tag information such as dish names and preparation processes from the video segments to be processed through a preset video analysis model, the efficiency of video analysis can be improved. Furthermore, when a user triggers a query request, the method can obtain the corresponding display video based on the monitoring video and tag information and output it to the user. This satisfies various user query needs while improving video query efficiency and, to a certain extent, ensuring the validity of the output display video.

[0060] At the same time, this method can significantly improve restaurant operation and management efficiency, the intelligence level of video analysis, and the user experience level.

[0061] Optionally, after the above-described operation of obtaining the tag information corresponding to the video segment to be processed, the embodiments of the present invention may further include:

[0062] S21. Determine any dish name as the name to be displayed, and based on the tag information corresponding to each video segment to be processed, determine the video segment to be processed whose dish name in the tag information matches the name to be displayed as the video segment to be edited.

[0063] S22. Based on the production process in the tag information corresponding to each of the video segments to be edited, sort the video segments to be edited according to the preset dish production process to obtain the sorting result.

[0064] S23. According to the sorting result, the video segments to be edited are spliced ​​together to obtain the video to be displayed corresponding to the name to be displayed, and the video to be displayed is added to the preset video library.

[0065] Specifically, for any one of the dish names corresponding to all the video segments to be processed, it can be determined as the name to be displayed. Based on the tag information corresponding to each video segment to be processed, the video segments whose dish names in the tag information match the name to be displayed are determined as video segments to be edited. Specifically, one dish name corresponds to a group of video segments to be edited, and all dish names in a group of video segments to be edited are identical. Therefore, video segments corresponding to the same dish can be grouped into the same group of video segments to be edited.

[0066] The aforementioned food preparation process can be pre-set. Specifically, since the order of each step in the food preparation process is often fixed, this embodiment of the invention can sort the video segments to be edited according to the preset food preparation process to obtain a sorting result. This sorting result can display the preparation process of the dish corresponding to the name to be displayed.

[0067] Specifically, the above-mentioned food preparation process can be: taking the ingredients, preparing the ingredients, cooking, and serving. Of course, different food preparation processes can be set up or other preparation steps can be added according to different dishes. This embodiment of the invention does not limit this.

[0068] Furthermore, the video segments to be edited can be spliced ​​together according to the sorting result to obtain the video to be displayed corresponding to the name to be displayed, and the video to be displayed can be added to the preset video library.

[0069] Optionally, during the above splicing process, if there are multiple video segments to be edited for the same production stage, only one video segment can be selected for splicing. Furthermore, to avoid the video to be displayed being too long, a video segment of a preset duration can be selected from the video segments to be edited corresponding to a production stage, thereby splicing video segments corresponding to different production stages. For example, the preset duration can be 5s, 6s, 7s, etc., and correspondingly, the video to be displayed corresponding to a name can be 20s, 24s, 28s, etc., which can be set according to actual needs; this embodiment of the invention does not impose any limitations on this.

[0070] Furthermore, the videos to be displayed can be added to a preset video library, thereby achieving structured video construction and facilitating subsequent video search operations. Specifically, the name of the video to be displayed can be used as a unique identifier, thus facilitating video retrieval.

[0071] In this way, embodiments of the present invention can construct videos to be displayed to demonstrate the preparation process for different dishes, which can be used to meet users' dish search needs and improve the effectiveness and diversity of video search.

[0072] Optionally, the operation of obtaining the display video corresponding to the query request based on the tag information corresponding to the monitoring video or each of the video segments to be processed may specifically include the following in this embodiment of the invention:

[0073] S31. If the query request is a dish process query request, the dish name indicated by the query request shall be determined as the target dish.

[0074] S32. Select a video from the preset video library that corresponds to the target dish as the video to be displayed in the query request.

[0075] In this embodiment of the invention, the aforementioned dish process query request is used to indicate the cooking process of a specific dish. Therefore, the dish name indicated by the query request can be identified as the target dish. Specifically, the query request may include a first query vector and a second query vector. The first query vector indicates the name of the dish to be queried, and the second query vector indicates the cooking process. Based on a large-scale model semantic matching and feature vector retrieval algorithm, the second query vector determines that the query request belongs to a dish process query request, and the first query vector determines the target dish.

[0076] Furthermore, the target dish can be traversed in a preset video library, and the video to be displayed corresponding to the target dish can be selected as the display video for the query request.

[0077] This allows for quick location of the display video that matches the query request within the pre-defined video library, achieving efficient response.

[0078] Optionally, if the video to be displayed corresponding to the target dish does not exist in the preset video library, the video to be displayed corresponding to the target dish can be regenerated based on the original monitoring videos and the video segments to be processed contained in the preset video library. Specifically, the target dish can be used as the name to be displayed, and the operations of steps S21 to S23 above can be performed.

[0079] Optionally, the tag information further includes the monitoring time; after the above-described operation of obtaining the tag information corresponding to the video segment to be processed, the embodiments of the present invention may further include:

[0080] S41. Set the preset editing parameters of the video segments to be processed that belong to the preset time period as the target parameters.

[0081] The above-mentioned operation of determining the video segment to be processed as the video segment to be edited if the dish name in the label information matches the name to be displayed can be further included in the following specific embodiments of the present invention:

[0082] S42. The video segment to be processed that matches the dish name in the label information with the name to be displayed and whose preset editing parameters are the target parameters is determined as the video segment to be edited.

[0083] The aforementioned monitoring time can be the video recording time, used to characterize the time when the video content occurred. The aforementioned preset time period can be set by the user, for example, it can be business hours or a time period with high customer traffic. The aforementioned preset editing parameters are used to characterize whether the video segment to be processed is allowed to be edited, and correspondingly, the target parameters are used to characterize whether editing is allowed.

[0084] For example, the preset editing parameters mentioned above can be activation parameters, and the target parameters can be 1 or true. Correspondingly, the preset editing parameters mentioned above can also be default parameters, which can be 0 or false. Accordingly, the default parameters are used to indicate that editing is not allowed.

[0085] Accordingly, the above operation of determining the video segment to be edited can be: determining the video segment to be processed that matches the dish name in the tag information with the name to be displayed and whose preset editing parameters are the target parameters as the video segment to be edited.

[0086] Specifically, the amount of data processing can be reduced to some extent by setting preset time periods to filter video segments to be processed.

[0087] Optionally, the aforementioned tag information may also include personnel status, which may include the working status of restaurant staff, and may also include the status of customers, which may include dining, waiting, serving, etc., and this embodiment of the invention does not impose any limitations on this. The aforementioned tag information may also include scene classification, used to characterize the monitored restaurant area.

[0088] Optionally, the embodiments of the present invention may further include setting the preset editing parameters of the video segments to be processed whose dish names belong to a preset dish set as default parameters.

[0089] Specifically, the preset menu can be set according to the actual sales needs of the restaurant to be monitored. It can include the names of dishes with high sales volume, or it can be set according to the restaurant menu. This embodiment of the invention does not limit this.

[0090] Optionally, the preset editing parameters for the video segments corresponding to dish names that do not belong to the preset dish set can be set to default parameters, so that only dishes belonging to the preset dish set can be created for display. This reduces the workload of video processing and ensures the utilization rate of the videos to be displayed.

[0091] Optionally, after obtaining the video segment to be processed through step 101 above, the preset slice parameters of each video segment to be processed can be set as target parameters, thereby identifying the video segment to be processed with both the preset slice parameters and the preset editing parameters as target parameters as the video segment to be edited. The preset slice parameters are used to indicate whether the video segment to be processed has undergone keyframe extraction processing, and correspondingly, the target parameters are used to indicate that it has undergone keyframe extraction processing.

[0092] This ensures, to a certain extent, that the determined video segment to be edited contains monitoring video content within a preset time period, thus meeting the user's monitoring needs and avoiding a large amount of invalid time period information in the video segment to be edited, thereby improving the effectiveness of the displayed video.

[0093] Optionally, after determining the video segment to be processed that matches the dish name in the label information with the name to be displayed as the video segment to be edited, the method further includes:

[0094] S51. Based on the video features of each of the video segments to be edited, generate subtitles and dubbing for each of the video segments to be edited.

[0095] S52. Add the subtitles and voice-overs of each of the video segments to be edited to each of the video segments to be edited.

[0096] The aforementioned video features refer to the feature information that characterizes the video content. Step S51 can be implemented by a pre-trained video understanding model. Each video segment to be edited can be input into the video understanding model. The video understanding model determines the video features through the images of the video segments to be edited, and then generates descriptive text for each video segment to be edited.

[0097] Furthermore, the descriptive text can be used as subtitles for the video segment to be edited, and the dubbing for the video segment to be edited can be obtained by converting the descriptive text into audio data.

[0098] Optionally, the subtitles and voice-over can be generated at preset intervals based on the duration of the video segment to be edited. For example, the interval can be 9 seconds, or it can be 10 seconds, 7 seconds, etc., and this embodiment of the invention does not limit this.

[0099] Furthermore, subtitles and voiceovers can be added to each video segment to be edited. These video segments can also be added to a preset video library, enabling modular segmented storage for different production stages.

[0100] This can improve the diversity and effectiveness of video presentations.

[0101] Optionally, when the query request is for a dish's process, a video containing complete visuals, subtitles, and narration can be directly output. Correspondingly, when the query request is for a specific step in a dish's preparation, a corresponding video segment can be selected from the video segments to be edited, based on the target dish and preparation step indicated in the query request, to achieve independent display of the preparation step.

[0102] Optionally, if the above query request is a general monitoring request, the original monitoring video or other video segments to be processed can be retrieved from the preset video library for display.

[0103] It should be noted that restaurants currently commonly use traditional video surveillance systems. These systems typically only have basic recording and storage functions, primarily used for security and post-incident review. Their video data is stored as long, continuous streams, lacking structured processing methods, resulting in low content retrieval efficiency and difficulty in target location. In actual operation, when managers or customers need to view the preparation process of a dish, the operational status during a specific time period, or the occurrence of a particular event, they often need to manually drag the timeline and repeatedly browse lengthy video content, which is not only time-consuming and laborious but also cannot guarantee accurate retrieval.

[0104] In addition, traditional systems lack semantic understanding and content tagging capabilities, making it impossible to automatically identify and classify dish types, processing procedures, personnel actions, scene states, etc. Therefore, it is difficult to achieve accurate recall and interactive display of specific information, which seriously restricts the value of video data in operational analysis, process optimization, customer interaction, and other aspects.

[0105] As the catering industry moves towards digitalization and intelligentization, restaurant managers are placing higher demands on video data. They not only need to retrospectively analyze the preparation process of important dishes but also expect to quickly generate structured, visualized video summaries for training, promotion, or service demonstrations. Meanwhile, customers also have personalized interactive needs, such as wanting to quickly view "how my ordered dish was made" on large screens or mini-programs. Faced with these new usage scenarios, traditional monitoring systems are severely lacking in intelligent analysis, precise retrieval, and real-time interaction capabilities. There is an urgent need for a new solution that combines deep learning, large-scale model semantic understanding, structured editing, and human-computer interaction to achieve in-depth value mining and efficient service support for restaurant video resources.

[0106] Based on this, Figure 2 This is a flowchart illustrating a real-scene video intelligent analysis and interaction method in a catering service scenario according to an embodiment of the present invention. Figure 2 As shown, it specifically includes the following four stages:

[0107] a. Raw Video Acquisition and Preprocessing: First, continuous raw video data recorded by the restaurant's monitoring equipment is acquired. The video is then intelligently compressed and keyframes extracted to reduce data redundancy and retain the most informative video clips for subsequent analysis and processing. This step lays the foundation for the construction and efficient use of the video library.

[0108] b. Video Content Recognition and Structured Tag Generation: A pre-trained video understanding model based on deep learning is used to perform refined recognition of compressed video segments, identifying key content such as specific dishes, scenes, and personnel operation states. Through this step, the system generates structured tags for each video segment, containing metadata such as "dish name," "timestamp activation," and "slice activation."

[0109] c. User Intent Understanding and Precise Video Content Retrieval: By analyzing user query intent through natural language understanding, and using large-scale model retrieval technology combined with feature vector retrieval algorithms, precise video content search is achieved.

[0110] Specifically, users input query requests via text or voice. The system uses a natural language processing module to semantically parse the instructions and construct a query vector. Combining large-scale pre-trained models (such as the GPT series) with semantic matching and feature vector retrieval algorithms, the system can quickly locate video clips or pre-made videos that match the query content in the video database, achieving efficient response.

[0111] d. Video Pre-production and Human-Computer Interaction Return: Automatically generates video clips with subtitles and voice-overs, as well as pre-edited videos, and returns the corresponding videos based on human-computer interaction instructions.

[0112] Specifically, the system can automatically generate pre-made videos (such as complete process or key segment clips) with subtitles and voiceovers from matched video clips, and overlay a light-colored watermark on the video to ensure copyright protection. Then, based on the human-computer interaction results, the system returns the matched video link, summary description, or pre-made video clip to the user, achieving a structured and interactive video experience.

[0113] Furthermore, Figure 3 This is a schematic diagram of label information in an embodiment of the present invention, such as... Figure 3As shown, this embodiment of the invention categorizes dishes into specific dishes and general dishes. For specific dishes (such as "braised fish"), the system marks them as having a clear dish name and sets both the "timestamp activation flag" and the "compressed slice activation flag" to True, indicating that the segment has been confirmed to be related to the dish's preparation process and will be used for subsequent editing and display. For general dishes, the dish name is empty or "none," only the compressed slice activation flag is set to True, and the timestamp activation flag is set to False, indicating that although the segment is compressed and stored, it does not participate in the dish's process analysis. This structure is used to support the system's intelligent filtering, classification, and precise retrieval of video content. The aforementioned specific dishes can be dishes belonging to the aforementioned preset dish set, and correspondingly, general dishes can be dishes that do not belong to the aforementioned preset dish set.

[0114] Specifically, the aforementioned tag information can be in a lightweight data exchange format (JavaScript Object Notation, JSON). A JSON file can be generated for each video segment to be processed.

[0115] Optionally, Figure 4 This is a schematic diagram of the overall processing flow of the restaurant video intelligent analysis system in an embodiment of the present invention, as shown below. Figure 4 As shown:

[0116] First, the system continuously collects raw video streams, including those depicting food preparation, cooking, and serving, and stores them in a video library. Then, the system performs intelligent compression and keyframe extraction on the raw video, generating more compact, frame-extracted videos (video segments to be processed) that are easier to analyze and process later.

[0117] Next, a pre-trained video understanding model (video analysis model) is used to perform semantic parsing on the extracted video frames, identifying whether each segment contains actions or scenes related to food preparation. Based on the recognition results, the system divides the video segments into two categories: specific dishes and general dishes, and generates a corresponding JSON file for each dish, labeling key metadata such as dish name, activation status, and time information.

[0118] Optionally, Figure 5 This is a schematic diagram of the overall processing flow of the restaurant video intelligent analysis system in an embodiment of the present invention, as shown below. Figure 5 As shown, users can efficiently obtain the restaurant surveillance video content they need by inputting natural language commands.

[0119] First, the user inputs a query command via text or voice. The system then processes this input to understand the user's intent, determining whether the user wants to view a general monitoring environment or a specific food preparation process, including:

[0120] If the user intends to view the monitored environment, the system will directly call up the original video or retrieve relevant clips from the video library to provide real-time or historical monitoring footage.

[0121] If the user intends to view the food preparation process, the system will search for the corresponding dish in the video library. If a matching pre-made video (the video to be displayed) already exists, it will be displayed directly; otherwise, the system will backtrack to the original video to construct a matching video or indicate that no matching result exists.

[0122] During the video pre-production stage, for each specific dish, the system splices together compressed video clips according to its preparation process (such as four stages: ingredient retrieval, ingredient preparation, cooking, and serving). Approximately 5 seconds of video are selected from each stage, resulting in a total of about 20 seconds of video showing the dish's preparation process. This process transforms the original compressed video footage into coherent short video segments, constructing structured content units.

[0123] After automated pre-production, to ensure the accuracy and comprehensibility of the generated content, the system generates a concise, objective subtitle and voice-over description every 9 seconds. This information is then fed to the video understanding model to describe the current video content. Finally, based on the subtitles and the original video, two types of pre-produced content are synthesized:

[0124] A pre-made version of the overall process video, including full footage, subtitles, and voice-over;

[0125] Pre-made versions of segmented compressed clips are available for later modular playback or independent use.

[0126] Thus, this invention, through video compression, deep learning model understanding, multi-dimensional feature analysis, and natural language processing technologies, can effectively improve the accuracy of video analysis, content retrieval efficiency, and user experience. It significantly enhances restaurant operation and management efficiency, the intelligence level of video analysis, and the overall user experience.

[0127] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0128] Figure 6 This is a schematic diagram of the structure of a real-scene video intelligent analysis and interaction system in a catering service scenario according to an embodiment of the present invention, such as... Figure 6 As shown, the system may include:

[0129] The first acquisition module 201 is used to acquire key video frames from at least one monitoring video of the restaurant to be monitored, and obtain at least one video segment to be processed.

[0130] The second acquisition module 202 is used to acquire, based on a preset video analysis model, the tag information corresponding to any of the video segments to be processed; the tag information includes at least the dish name and the preparation process.

[0131] The third acquisition module 203 is used to respond to the user's query request, acquire the display video corresponding to the query request based on the tag information corresponding to the monitoring video or each of the video segments to be processed, and output the display video.

[0132] Optionally, the system also includes:

[0133] The first determining module is used to determine any dish name as the name to be displayed, and to determine the video segment to be processed that has the same dish name in the tag information as the name to be displayed as the video segment to be edited according to the tag information corresponding to each video segment to be processed.

[0134] The sorting module is used to sort the video segments to be edited according to the production steps in the tag information corresponding to each video segment to be edited, and according to the preset dish production process, to obtain the sorting result;

[0135] The splicing module is used to splice the video segments to be edited according to the sorting result to obtain the video to be displayed corresponding to the name to be displayed, and add the video to be displayed to the preset video library.

[0136] Optionally, the third acquisition module includes:

[0137] The second determining submodule is used to determine the dish name indicated by the query request as the target dish if the query request belongs to the dish process query request;

[0138] The selection submodule is used to select a video corresponding to the target dish from the preset video library as the display video corresponding to the query request.

[0139] Optionally, the tag information also includes monitoring time; the system also includes:

[0140] The settings module is used to set the preset editing parameters of the video segments to be processed that are monitored within a preset time period as the target parameters.

[0141] The first determining module is specifically used for:

[0142] The video segment to be edited is the one whose dish name in the label information matches the name to be displayed and whose preset editing parameters are the target parameters.

[0143] Optionally, the system also includes:

[0144] The generation module is used to generate subtitles and dubbing for each of the video segments to be edited based on the video features of each video segment to be edited.

[0145] An add module is used to add subtitles and voice-overs to each of the video segments to be edited.

[0146] Optionally, the system also includes:

[0147] The receiving module is used to receive the user's query command;

[0148] The parsing module is used to perform semantic parsing on the query command to obtain a query request that matches the user's query intent.

[0149] Optionally, the third acquisition module includes:

[0150] The text generation submodule is used to generate summary text for the display video based on the display video;

[0151] The output submodule is used to output the summary text and the display video.

[0152] In summary, the real-time video intelligent analysis and interaction system for catering service scenarios provided by this invention obtains at least one video segment to be processed by acquiring key video frames from at least one monitoring video of the restaurant to be monitored. For any one of the video segments to be processed, the system obtains the corresponding tag information based on a preset video analysis model. The tag information includes at least the dish name and the preparation process. In response to a user's query request, the system obtains the display video corresponding to the query request based on the monitoring video or the tag information corresponding to each of the video segments to be processed, and outputs the display video. Thus, by acquiring key video frames from the monitoring video of the restaurant to be monitored, this invention can retain the most informative video segments to be processed. Simultaneously, by obtaining tag information such as dish names and preparation processes from the video segments to be processed through a preset video analysis model, the system can improve video analysis efficiency. Furthermore, when a user triggers a query request, the system can obtain the corresponding display video based on the monitoring video and tag information, thereby satisfying various user query needs, improving video query efficiency, and ensuring the validity of the output display video to a certain extent.

[0153] For the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0154] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0155] Regarding the real-scene video intelligent analysis and interaction system for catering service scenarios in the above embodiments, the specific methods of each module's operation have been described in detail in the embodiments of the relevant method, and will not be elaborated here.

[0156] This invention also provides an electronic device, including: a processor and a memory for storing processor-executable instructions, wherein the processor is configured to execute the above-described method for intelligent analysis and interaction of real-scene video in a catering service scenario.

[0157] Reference Figure 7 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Figure 7 As shown, electronic device 1200 includes at least one processor 1201, memory 1202, user interface 1203, and at least one network interface 1204. The various components in electronic device 1200 are coupled together via bus system 1205. It can be understood that bus system 1205 is used to implement communication between these components. In addition to a data bus, bus system 1205 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general labeled all buses as Bus System 1205.

[0158] The user interface 1203 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0159] It is understood that the user interface 1203 is a human-computer interaction unit used for human-computer interaction operation of electronic devices. The electronic device 1200 also includes an image capturing unit for capturing images, and can thus be used as a capturing device.

[0160] The memory 1202 in this embodiment is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0161] The image processing method and / or video editing method disclosed in the embodiments of this application can be applied to, or implemented by, processor 1201. Processor 1201 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the image processing method can be completed by the integrated logic circuitry in the hardware of processor 1201 or by instructions in software form. The processor 1201 described above can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 1201 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory 1202. The processor 1201 reads the information in memory 1202 and, in conjunction with its hardware, completes the steps of the image processing method and / or video editing method provided in the embodiments of this application.

[0162] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0163] It is understood that memory 1202 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Sync Link Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0164] In an exemplary embodiment, this application also provides a computer storage medium, specifically a computer-readable storage medium, such as a memory 1202 storing a computer program. This computer program can be executed by the processor 1201 of the electronic device 1200 to complete the steps described in the method of this application embodiment. The computer-readable storage medium can be a ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0165] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor 1201 of an electronic device 1200 to perform the steps described in the method of this application embodiment.

[0166] This invention also provides a non-transitory computer-readable storage medium that, when instructions in the storage medium are executed by a processor of an electronic device (server or terminal), enables the processor to perform... Figure 1 The method for intelligent analysis and interaction of real-scene video in the catering service scenario is shown.

[0167] This application also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described embodiments of the intelligent analysis and interaction method for real-scene video in the catering service scenario, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0168] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0169] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0170] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, systems, or computer program products. Therefore, embodiments of the present invention can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0171] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0172] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing terminal device to operate in a predictive manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0174] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0175] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0176] It should be noted that the various data-related processes in the embodiments of this application are carried out in compliance with the relevant data protection laws and policies of the country where the location is located, and with the authorization granted by the owner of the corresponding device.

[0177] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0178] The above provides a detailed description of the intelligent analysis and interaction method, system, electronic device, and medium for real-scene video in a catering service scenario provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for intelligent analysis and interaction of real-scene video in a catering service scenario, characterized in that, The method includes: Based on at least one surveillance video of the restaurant to be monitored, key video frames are obtained from the surveillance video to obtain at least one video segment to be processed. For any of the video segments to be processed, based on a preset video analysis model, the tag information corresponding to the video segment to be processed is obtained; the tag information includes at least the dish name and the preparation process; In response to a user's query request, based on the tag information corresponding to the monitoring video or each of the video segments to be processed, the display video corresponding to the query request is obtained and the display video is output.

2. The method according to claim 1, characterized in that, After obtaining the tag information corresponding to the video segment to be processed, the method further includes: Any dish name is determined as the name to be displayed, and based on the tag information corresponding to each video segment to be processed, the video segment whose dish name in the tag information matches the name to be displayed is determined as the video segment to be edited; Based on the production process in the tag information corresponding to each of the video segments to be edited, the video segments to be edited are sorted according to the preset dish production process to obtain the sorting result; The video segments to be edited are spliced ​​together according to the sorting result to obtain the video to be displayed corresponding to the name to be displayed, and the video to be displayed is added to the preset video library.

3. The method according to claim 2, characterized in that, The step of obtaining the display video corresponding to the query request based on the tag information corresponding to the monitoring video or each of the video segments to be processed includes: If the query request is a dish process query request, the dish name indicated by the query request is determined as the target dish; From the preset video library, select the video to be displayed that corresponds to the target dish, and use it as the display video corresponding to the query request.

4. The method according to claim 2, characterized in that, The tag information also includes monitoring time; after obtaining the tag information corresponding to the video segment to be processed, the method further includes: Set the preset editing parameters of the video segments to be processed that belong to the preset time period as the target parameters; The step of identifying video segments whose dish names in the label information match the names to be displayed as video segments to be edited includes: The video segment to be edited is the one whose dish name in the label information matches the name to be displayed and whose preset editing parameters are the target parameters.

5. The method according to claim 2, characterized in that, After determining the video segment to be processed that matches the dish name in the label information with the name to be displayed as the video segment to be edited, the method further includes: Based on the video features of each of the video segments to be edited, subtitles and dubbing are generated for each of the video segments to be edited. Add the subtitles and voiceovers to each of the video segments to be edited.

6. The method according to claim 1, characterized in that, The method further includes: Receive the user's query command; The query instruction is semantically parsed to obtain a query request that matches the user's query intent.

7. The method according to any one of claims 1-6, characterized in that, The output of the displayed video includes: Based on the demonstration video, generate summary text for the demonstration video; Output the summary text and the demonstration video.

8. A real-scene video intelligent analysis and interaction system for catering service scenarios, characterized in that, The system includes: The first acquisition module is used to acquire key video frames from at least one monitoring video of the restaurant to be monitored, and obtain at least one video segment to be processed. The second acquisition module is used to acquire the tag information corresponding to any of the video segments to be processed based on a preset video analysis model; the tag information includes at least the dish name and the preparation process. The third acquisition module is used to respond to the user's query request, acquire the display video corresponding to the query request based on the tag information corresponding to the monitoring video or each of the video segments to be processed, and output the display video.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images