Method and device for processing video, and household appliance

Abstract

The present application relates to the field of cooking technology, and discloses a method for processing a video, the method comprising: parsing a video to extract feature information, and identifying the feature information; when the recognition result contains setting content, marking the time point according to the setting content, and generating interpolated video information based on the setting content; adding the corresponding interpolated video information at the time point corresponding to the mark. In the disclosed embodiment, the video feature information is parsed, the feature information is identified, and when the setting content corresponding to the mark is included, the video is dotted and marked, and interpolation processing is performed based on the setting content and the time point corresponding to the mark, and the video is divided into multiple continuously playable segments. The user can quickly view the content of the specified segment through the mark, which is convenient for the user to accurately control the video playback and the segments related to the cooking steps during the cooking process, and can improve the cooking effect. The application also discloses a device and household appliance for processing a video.

Description

Technical Field

[0001] The present application relates to the field of cooking technology, for example, to a method and device for processing videos, and household appliances. Background Art

[0002] At present, with the improvement of living standards, people's pursuit of delicious food is also increasing. More people are willing to experience the fun of cooking at home and have shared the cooking process through videos. In the process of making delicious food, recipes and their methods are often required as references, such as manuscripts or audio and video. From the user's usage habits, text or voice descriptions are far less good than video guidance. In the prior art, users can play videos and control the playback progress through various playback devices, and guide users on how to cook based on recipe-related videos.

[0003] In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art:

[0004] During the cooking process, users cannot accurately control the video playback segment, which affects the cooking effect. Summary of the invention

[0005] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0006] The embodiments of the present disclosure provide a method and apparatus for processing videos, and a household appliance, so as to solve the technical problem that videos have no unified format due to different video sources, and users cannot accurately control the video playback segments during the cooking process, which affects the cooking effect.

[0007] In some embodiments, the method comprises:

[0008] Analyze the video to extract feature information and identify the feature information;

[0009] When the recognition result contains set content, the time point is marked according to the set content;

[0010] The video is interpolated according to the marker.

[0011] In some embodiments, the device includes: a processor and a memory storing program instructions, and the processor is configured to perform the above-mentioned method for processing a video when executing the program instructions.

[0012] In some embodiments, the household appliance includes the above-mentioned device for processing video.

[0013] The method and apparatus for processing video, and home appliance provided by the embodiments of the present disclosure can achieve the following technical effects:

[0014] Analyze the video feature information, identify the feature information, and mark the video when it contains set content corresponding to the mark. Perform frame insertion based on the set content and the time point corresponding to the mark, and divide the video into multiple continuously playable segments. Users can quickly view the content of the specified segment through marking, which is convenient for users to accurately control the video playback and the segments related to the cooking steps during the cooking process, which can improve the cooking effect.

[0015] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0017] Figure 1 is a schematic diagram of a method for processing a video provided by an embodiment of the present disclosure;

[0018] Figure 2 is a schematic diagram for marking time points provided by an embodiment of the present disclosure;

[0019] Figure 3 is another schematic diagram for marking time points provided by an embodiment of the present disclosure;

[0020] Figure 4 It is an interactive interface for obtaining food information provided by the embodiment of the present disclosure;

[0021] Figure 5 It is an interactive interface for obtaining taste provided by the embodiment of the present disclosure;

[0022] Figure 6 It is an interactive interface in the form of a menu acquisition provided by the embodiment of the present disclosure;

[0023] Figure 7 It is an interactive interface for displaying text recipe materials provided by an embodiment of the present disclosure;

[0024] Figure 8 It is a schematic diagram of a device for processing video provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0026] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0027] Unless otherwise stated, the term "plurality" means two or more.

[0028] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0029] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0030] The inability to accurately control the video playback progress to guide the current cooking steps during the cooking process will result in unsatisfactory cooked dishes.

[0031] Figure 1 is a schematic diagram of a method for processing a video provided by an embodiment of the present disclosure, the method comprising the following steps:

[0032] S101, parsing the video to extract feature information, and identifying the feature information.

[0033] S102: When the recognition result includes setting content, mark the time point according to the setting content, and generate interpolated frame video information based on the setting content.

[0034] S103, adding corresponding interpolated frame video information at the time point corresponding to the mark.

[0035] In the embodiment of the present disclosure, the video processing device for executing the above method is a mobile terminal or a smart home appliance with a video playback function, such as a refrigerator, a television, or a range hood, etc. The embodiment of the present application mainly takes a refrigerator as an example to illustrate the solution.

[0036] In step S101, the feature information includes one or more of subtitle information, sound information, and image information. Depending on the video producer, the content included in the video is different. In some embodiments, the video producer dubs the video and adds subtitles. In some embodiments, the video producer dubs the video without adding subtitles. In some embodiments, the video producer adds subtitles to the video and adds background music to the video or does not dub. Therefore, depending on the video production, the feature information extracted from the video by the video processing device is different.

[0037] In some embodiments, the video producer will not only explain the specific operation steps when dubbing the video, but also add background music. When obtaining sound information, the video processing device mainly separates the audio track and obtains the user's sound information.

[0038] In different embodiments, when the feature information contained in the video is different, the method of identifying the feature information is different.

[0039] In some embodiments, in step S101, identifying the feature information includes: when the video includes multiple feature information, identifying each of the multiple feature information. After identifying the multiple feature information, the video processing device can correct the information with identification differences based on the identification results of the multiple feature information. For example, when the dubbing is unclear or wrong, the correct recipe information can be determined by combining the subtitles and dubbing.

[0040] In some embodiments, in step S101, identifying the feature information includes: when the video includes multiple feature information, identifying the feature information with the highest priority. The video processing device only identifies one type of feature information, which can shorten the video identification time and improve the video processing efficiency.

[0041] In some embodiments, when the user does not add subtitles or dubbing to the video, the video processing device identifies the set content based on the image information.

[0042] In some embodiments, the priority of subtitle information is higher than that of sound information; the priority of sound information is higher than that of image information. Subtitle information more intuitively displays recipe information, and has high recognition efficiency and is less prone to errors. Sound information can reduce recognition efficiency due to the accent of the video producer, the presence of noise in the dubbing environment, and other reasons.

[0043] In some embodiments, identifying the feature information includes one or more of the following: when the feature information is subtitle information, performing text recognition on the subtitle information and converting it into text information; when the feature information is sound information, performing semantic recognition on the sound information and converting it into text information; when the feature information is image information, performing image recognition on multiple consecutive frames of images to determine the action of the target object and converting it into text information.

[0044] In some embodiments, if the video does not contain subtitle information, the sound information in the video is directly parsed. Different words correspond to different audio signal frequencies and amplitudes, so the sound information is converted into text information by parsing the audio signal.

[0045] In some embodiments, the method further includes: proofreading and correcting the text information, and using the corrected text information as the recognition result. In particular, when the video processing device recognizes sound information, the recognized text information may contain grammatical or phrase errors due to the accent of the video producer, the presence of noise in the dubbing environment, etc. The video processing device proofreads and corrects the text information, which can improve the accuracy of the recognition result.

[0046] In some embodiments, the setting content in step S102 includes keywords and / or setting actions.

[0047] Optional keywords include first step, second step,…, nth step, then, 1, 2,…, n, 2 minutes, 5 minutes,…, n minutes, turn on the fire, add fuel, turn off the fire, etc.

[0048] Optionally, set actions include turning on the fire, adding oil, adding salt, stir-frying, turning off the fire, etc.

[0049] After the video processing device determines that the action is firing based on the multi-needle image information, the identified action is recorded as text information and the time point corresponding to the action is recorded.

[0050] In different embodiments, the time points are marked in different ways according to the setting content.

[0051] Figure 2 This is a schematic diagram of marking time points provided by an embodiment of the present disclosure. For videos with subtitle information, the subtitle file is automatically obtained, and then the subtitle information is retrieved. After the keyword information is identified, it is converted into the corresponding time position information. In order to facilitate the user to use the wake-up word, the corresponding keyword position can be quickly called and switched, and the video is pre-processed by inserting frames. The corresponding inserting frame position is used as the software call dot mark position.

[0052] Optionally, the user can control the video playback through voice, options, buttons, etc. In the cooking video playback mode, the main voice interaction wake-up words include previous step, next step, nth step, turn on the fire, add oil, add salt, play, pause, fast forward, fast rewind, etc. The disclosed embodiment can effectively avoid the user's distress of repeatedly re-operating or switching playback due to not being able to keep up with the speed of the video tutorial during the food making process.

[0053] Figure 3 It is another schematic diagram for marking time points provided by an embodiment of the present disclosure. In some embodiments, if the video does not contain subtitle information, the sound information in the video is directly parsed. The audio signal frequency and amplitude corresponding to different texts are different. Therefore, the sound information is converted into text information by parsing the audio signal, and then the text information is retrieved. After the keyword information is identified, the original video is segmented into ordinary audio segments and keyword audio segments. To facilitate users to use wake-up words, you can quickly call and switch the corresponding keyword position, perform interpolation preprocessing on the video, and use the corresponding interpolation position as the software call dot marker.

[0054] In some embodiments, in step S102, generating interpolated video information based on the set content includes: rendering the text information corresponding to the keyword and / or the set action to generate video text content; obtaining preset image information, and superimposing the preset image information with the video text content to generate interpolated video information. The user can control the video playback progress based on the keyword corresponding to the interpolated information.

[0055] In some embodiments, one or more of a font, texture, size, style, and color of the text information is rendered.

[0056] In different embodiments, the preset image information is determined in different ways.

[0057] Optionally, the preset image information is determined according to the set action included in the step, so that the user can more intuitively determine the specific operation information corresponding to the current step. For example, the preset image information corresponding to the firing action is a knob image, an image marked with "open", or an image marked with "on". The preset image information corresponding to the refueling action includes an oil can or oil droplets.

[0058] Optionally, the preset image information is determined according to the dish, for example, if the dish is scrambled eggs with tomatoes, the preset image information is a picture of scrambled eggs with tomatoes, or the preset image information is a picture of the raw materials, tomatoes and eggs.

[0059] Optionally, the preset image information is a solid color background image, wherein the color of the background image is determined according to the font color, and the color difference between the background image color and the font color is greater than a set value, so as to ensure that the user can clearly see the specific content of the interpolated video information.

[0060] In the disclosed embodiment, the video feature information is parsed, the feature information is identified, and when it contains set content corresponding to the mark, the video is marked, and frame insertion is performed based on the set content and the time point corresponding to the mark, and the video is divided into multiple continuously playable segments. The user can quickly view the content of the specified segment through the mark, which is convenient for the user to accurately control the video playback and the segment related to the cooking steps during the cooking process, and can improve the cooking effect.

[0061] In step S101, the video processing device must determine the required video before acquiring the video related to the recipe, and the specific determination must be based on the food information.

[0062] The methods of obtaining food information are different in different embodiments.

[0063] In some embodiments, the video processing device actively obtains the image information of the food in the refrigerator through the intelligent recognition mode, analyzes the image information to obtain the existing food, and makes video recommendations based on the existing food to ensure that the user's cooking of dishes can proceed smoothly. When the video processing device is a refrigerator, the image acquisition device of the refrigerator acquires the image of the food in the refrigerator. When the video processing device is a device other than a refrigerator, the image acquisition device of the refrigerator acquires the image of the food in the refrigerator and acquires the image of the food sent by the refrigerator.

[0064] In some embodiments, the video processing device obtains the food information through the manual input mode. The video processing device provides dish options, and the user manually inputs the ingredients available at home. The manual input mode provides users with a choice of options. Users can input food information according to their favorite dishes. The video processing device makes relevant recommendations based on the food information input by the user, ensuring that the recommended recipes are more in line with the user's needs.

[0065] Figure 4 This is an interactive interface for obtaining food information provided by the embodiment of the present disclosure. Users can add or delete dishes by clicking the "+" and "-" options on the left side of the screen. The dish information is displayed on the right side of the screen. Optionally, the dish information is displayed in a list or thumbnail format. Users can create different food libraries, name them, and store them separately to meet the cooking needs of multiple users, such as Figure 4 “My family’s vegetable basket” shown.

[0066] Furthermore, the video processing device also provides a taste selection option. Figure 5This is an interactive interface for obtaining taste provided by the embodiment of the present disclosure. The video processing device recommends relevant recipes according to the taste selected by the user. Optionally, the taste options include sweet, sweet and spicy, spicy, medium spicy, spicy, etc. According to the taste of different dishes, the user can customize the taste to ensure that the video processing device recommends recipes that meet the user's needs. For example: slightly spicy, very spicy, etc.

[0067] Figure 6 It is an interactive interface for obtaining a menu form provided by an embodiment of the present disclosure. The video processing device can not only obtain recipe-related videos, but also obtain text recipes. Optionally, the text recipes are pure text recipes or recipes in the form of a combination of pictures and texts. The video processing device provides a variety of recipe format options to meet the viewing habits of users, and uses text recipes to more intuitively display the steps of making dishes, thereby improving the efficiency of users in selecting recipes.

[0068] In some embodiments, when the recipe is a text recipe, the recipe is displayed using a pagination display method. After searching for a recipe, if the recipe is already paginated, it is directly quoted. If the recipe is not paginated, the recipe is optimized before being displayed. The processing method is as described in the above embodiment to mark the time point according to the text information. Specifically, it includes: retrieving step-by-step keywords and marking them according to the keywords; if the keywords are not retrieved, marking the mark according to the line break; the keywords include step n, sequence number n, next step, and then, etc.

[0069] Figure 7 It is an interactive interface for displaying text recipe materials provided by an embodiment of the present disclosure. After completing the recipe paging, the text information of a certain step or a certain page of the recipe is displayed on the left side of the middle part of the screen, and the corresponding picture information is displayed on the right side of the middle part of the screen, and the name of the dish is displayed in the title position in the upper half of the screen. Optionally, when playing a recipe, the recipe switching display is controlled by voice, options or buttons. Among them, the switching display is performed by previous page, next page, nth page, previous step, next step, nth step, etc.

[0070] In some embodiments, a music accompaniment function is added during the text recipe display process, and music can be played while the recipe is displayed to improve the user experience. Figure 7 As shown, the user turns on the music accompaniment function through the music option, and the system randomly plays background music. In certain embodiments, the user can select the music to be played through voice, for example: I want to listen to ***'s music, I want to listen to ***, etc.

[0071] The embodiment of the present disclosure further provides a device for processing a video, including a processor and a memory storing program instructions, wherein the processor is configured to execute the method for processing a video as described above when executing the program instructions.

[0072] Combination Figure 8 As shown, an embodiment of the present disclosure provides a device for processing a video, including a processor (processor) 800 and a memory (memory) 801. Optionally, the device may also include a communication interface (Communication Interface) 802 and a bus 803. The processor 800, the communication interface 802, and the memory 801 may communicate with each other through the bus 803. The communication interface 802 may be used for information transmission. The processor 800 may call the logic instructions in the memory 801 to execute the method for processing a video in the above embodiment.

[0073] In addition, the logic instructions in the memory 801 described above can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0074] The memory 801 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 800 executes the function application and data processing by running the program instructions / modules stored in the memory 801, that is, implements the method for processing video in the above embodiment.

[0075] The memory 801 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 801 may include a high-speed random access memory and may also include a non-volatile memory.

[0076] The present disclosure also provides a home appliance, including the above-mentioned device for processing video. In some embodiments, the home appliance is a refrigerator, air conditioner, television, range hood, etc. that can play videos.

[0077] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned method for processing a video.

[0078] An embodiment of the present disclosure provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned method for ...

[0079] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0080] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0081] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0082] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0083] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0084] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for processing a video, characterized in that: include: Determine videos related to the recipe based on the ingredient information; Analyze the video to extract feature information and identify the feature information; In the case where the recognition result contains setting content, marking the time point according to the setting content; the setting content includes keywords and / or setting actions; Rendering text information corresponding to the keywords and / or set actions to generate video text content; Acquire preset image information, and superimpose the preset image information with the video text content to generate interpolated frame video information; wherein the preset image information is determined by: determining according to a set action; or, determining according to a dish in a recipe; or, using a solid color background image, the color of the background image is determined according to the font color, and the color difference between the background image color and the font color is greater than a set value; Add corresponding interpolated frame video information at the time point corresponding to the mark, so that the user can control the video playback progress based on the keyword corresponding to the interpolated frame video information; The method further comprises: determining a related text recipe according to the food information; the text recipe comprises a pure text recipe or a recipe in the form of a combination of text and images; When the text recipe is a paged recipe, the text recipe is directly quoted for display; or, when the text recipe is not a paged recipe, the text recipe is paged before being displayed; The paging process of the text recipe includes: searching for step-by-step keywords in the text recipe and marking them according to the keywords; or marking the text recipe according to line breaks; and paging the text recipe according to the markings.

2. The method according to claim 1, characterized in that The feature information includes one or more of subtitle information, sound information and image information.

3. The method according to claim 2, characterized in that Identifying characteristic information includes: In the case where the video includes multiple types of feature information, the multiple types of feature information are identified respectively; or, When a video includes multiple types of feature information, the feature information with the highest priority is identified.

4. The method according to claim 3, characterized in that The priority of subtitle information is higher than that of sound information; the priority of sound information is higher than that of image information.

5. The method according to claim 2, characterized in that: Identifying feature information includes one or more of the following: In the case where the characteristic information is subtitle information, performing text recognition on the subtitle information and converting it into text information; When the feature information is sound information, semantic recognition is performed on the sound information to convert it into text information; When the feature information is image information, image recognition is performed on multiple consecutive frames of images to determine the action of the target object and convert it into text information.

6. The method according to any one of claims 1 to 5, characterized in that: Rendering of one or more of the font, texture, size, style, and color of text information.

7. A device for processing video, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to perform the method for processing a video according to any one of claims 1 to 6 when executing the program instructions.

8. A household appliance, characterized in that: Comprising the apparatus for processing video as claimed in claim 7.

Images