Auxiliary operation method and system based on video interaction

Abstract

The embodiment of the invention discloses an auxiliary operation method and system based on video interaction. The method comprises the steps of displaying a specification display request page, calling a photographing assembly to obtain a target video in response to triggering of a specification display request control in the specification display request page, performing image recognition on a plurality of video frames in the target video, determining a product operation requirement according to a recognition result, and displaying the product operation requirement according to the product operation requirement. And determining a target operation scene corresponding to the product operation demand according to a preset mapping relationship, determining a marking component and / or operation prompt information according to the target operation scene, and highlighting the marking component and / or operation prompt information in the photographing picture display page. According to the method, the target operation scene matched with the user operation behavior in the video is determined by performing video analysis on the video picture collected by the user side, and then the corresponding marking component and / or the operation prompt are matched for the target operation scene, so that the user operation is guided in a step-by-step disassembling manner, and the user understanding and operation difficulty is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and more specifically, to an auxiliary operation method and system based on video interaction. Background Art

[0002] The function of the instruction manual is to provide users with comprehensive guidance on how to use, install, maintain, or understand a specific product. Existing forms of instruction manuals include paper instruction manuals and electronic instruction manuals. Paper instruction manuals usually use two-dimensional exploded views to disassemble and explain the steps of the components of complex equipment, while the user actually faces a three-dimensional complete product, which makes it difficult for users to understand and operate. Electronic instruction manuals include picture instruction manuals and video instruction manuals. Picture instruction manuals are similar to paper instruction manuals. For video instruction manuals, it is often difficult for users to understand the full content of the video in one video playback process, and they often need to repeatedly compare the content of the equipment entity and the video instruction manual during operation, resulting in high operation difficulty. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide an auxiliary operation method and system based on video interaction to disassemble the user operation into the smallest granularity of the instruction manual understanding process, so as to achieve step-by-step disassembly, and by highlighting the marked components and / or the operation prompt information in the photography screen display page, reduce the user's understanding and operation difficulty.

[0004] In a first aspect, an auxiliary operation method based on video interaction is provided, and the method includes:

[0005] Display an instruction manual display request page, where the instruction manual display request page includes an instruction manual display request control;

[0006] In response to the instruction manual display request control being triggered, call a photography component to obtain a target video;

[0007] Perform image recognition on multiple video frames in the target video respectively;

[0008] Determine product operation requirements according to the recognition results of the multiple video frames;

[0009] Determine a target operation scenario corresponding to the product operation requirements according to a preset mapping relationship;

[0010] Determine marked components and / or operation prompt information according to the target operation scenario, where the marked components are product components that need to be introduced currently;

[0011] Highlight the marked components and / or the operation prompt information in the photography screen display page.

[0012] Second aspect, there is provided an auxiliary operating system based on video interaction, the system comprising:

[0013] A server configured to obtain a to-be-converted instruction manual, the instruction manual being a paper instruction manual and / or a non-interactive video instruction manual, and input the to-be-converted instruction manual into an artificial intelligence model to obtain instruction manual interpretation content;

[0014] A client configured to display an instruction manual display request page, the instruction manual display request page including an instruction manual display request control, and in response to the instruction manual display request control being triggered, call a photography component to obtain a target video, perform image recognition on a plurality of video frames in the target video respectively, determine a product operation requirement according to the recognition results of the plurality of video frames, determine a target operation scenario corresponding to the product operation requirement according to a preset mapping relationship, search for a marked component and / or operation prompt information corresponding to the target operation scenario from the instruction manual interpretation content, the marked component being a product component that needs to be introduced currently, and highlight the marked component and / or the operation prompt information on a photography screen display page.

[0015] Third aspect, there is provided an auxiliary operating device based on video interaction, the device comprising:

[0016] A first display module for displaying an instruction manual display request page, the instruction manual display request page including an instruction manual display request control;

[0017] A component calling module for, in response to the instruction manual display request control being triggered, calling a photography component to obtain a target video;

[0018] An image recognition model for performing image recognition on a plurality of video frames in the target video respectively;

[0019] A first determination module for determining a product operation requirement according to the recognition results of the plurality of video frames;

[0020] A second determination module for determining a target operation scenario corresponding to the product operation requirement according to a preset mapping relationship;

[0021] A third determination module for determining a marked component and / or operation prompt information according to the target operation scenario, the marked component being a product component that needs to be introduced currently;

[0022] A second display module for highlighting the marked component and / or the operation prompt information on a photography screen display page.

[0023] Fourthly, an electronic device is provided, including a memory and a processor. The memory is used to store one or more computer program instructions. Among them, the one or more computer program instructions are executed by the processor to implement the method described in the first aspect above.

[0024] Fifthly, a computer-readable storage medium is provided. A computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0025] Sixthly, a computer program product is provided, including a computer program / instructions. When the computer program / instructions are executed by a processor, the method described in the first aspect above is implemented.

[0026] The technical solution of the embodiment of the present invention is to display a request page for the instruction manual. In response to the triggering of the instruction manual display request control in the instruction manual display request page, the photography component is called to obtain a target video. Image recognition is performed on multiple video frames in the target video respectively. According to the recognition results of the multiple video frames, the product operation requirements are determined. According to the preset mapping relationship, the target operation scenario corresponding to the product operation requirements is determined. According to the target operation scenario, the marking component and / or the operation prompt information are determined. The marking component is the product component that needs to be introduced currently. The marking component and / or the operation prompt information are prominently displayed in the photography screen display page. In this embodiment, through video analysis of the video screen collected by the user terminal, the target operation scenario matching the user operation behavior in the video is determined, and then the corresponding marking component and / or operation prompt are matched for it. The above technical solution realizes disassembling the user operation into the smallest granularity of the instruction manual understanding process, so as to realize guiding the user operation step by step and reducing the user's understanding and operation difficulty. Description of the Drawings

[0027] Through the following description of the embodiments of the present invention with reference to the drawings, the above and other objects, features and advantages of the present invention will become clearer. In the drawings:

[0028] Figure 1 It is a schematic diagram of an auxiliary operating system based on video interaction according to an embodiment of the present invention;

[0029] Figure 2 It is a flowchart of an auxiliary operation method based on video interaction according to an embodiment of the present invention;

[0030] Figure 3 It is a schematic diagram of a request page for displaying an instruction manual according to an embodiment of the present invention;

[0031] Figure 4 It is a schematic diagram of a photography screen display page according to an embodiment of the present invention;

[0032] Figure 5Flowchart of the method for using the labeling model according to an embodiment of the present invention;

[0033] Figure 6 Schematic diagram of the auxiliary operation device based on video interaction according to an embodiment of the present invention;

[0034] Figure 7 Schematic diagram of the electronic device according to an embodiment of the present invention. Detailed implementation manners

[0035] The following describes the present application based on embodiments, but the present application is not limited to these embodiments. In the following detailed description of the present application, some specific details are described in detail. Those skilled in the art can fully understand the present application without the description of these details. In order to avoid obscuring the essence of the present application, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0036] In addition, those of ordinary skill in the art should understand that the drawings provided herein are for illustrative purposes only, and the drawings are not necessarily drawn to scale.

[0037] Unless the context clearly requires otherwise, words such as "including" and "comprising" in the entire application document should be interpreted as having an inclusive meaning rather than an exclusive or exhaustive meaning; that is, it is the meaning of "including but not limited to".

[0038] In the description of the present application, it should be understood that terms such as "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise stated, the meaning of "a plurality" is two or more.

[0039] For the solutions described in this specification and the embodiments, if they involve personal information processing, they will be processed on the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for performing a contract, etc.), and will only be processed within the specified or agreed scope. If the user refuses to process personal information other than the necessary information required for the basic functions, it will not affect the user's use of the basic functions.

[0040] Figure 1 Schematic diagram of the auxiliary operation system based on video interaction according to an embodiment of the present invention. As Figure 1 shown, the auxiliary operation system based on video interaction includes a user terminal 11, a server 12, and an operation terminal 13.

[0041] The user terminal 11 and the operation terminal 13 are both general terminals that can run auxiliary operation platform applications or small programs based on video interaction. The terminals can be smart phones, tablet computers, car computers, smart wearable devices, laptops, car computers or other types of mobile communication terminals. It can also be a mobile device terminal that uses advanced technology to provide users with immersive virtual or augmented reality experiences, such as augmented reality devices (AR devices), virtual reality devices (VR devices) and mixed reality devices (MR devices). In some possible implementations, the user terminal 11 and the operation terminal 13 can also be dedicated terminals with corresponding applications solidified in a dedicated integrated circuit. Among them, the open platform enables third-party developers to develop their own small programs based on the open platform by providing an application programming interface (API) to third-party developers. The small program can specifically be a program developed on the basis of the application in the platform and used to perform corresponding operations. Among them, the user terminal 11 is a terminal device held by a user who wishes to read the instructions, which has a photographic component that can display video images and / or virtual or augmented reality images. The operation terminal 13 is a terminal device held by the staff who pre-records the video material.

[0042] The server 12 refers to a general data processing device that can provide computing or application services for the charging service provider platform. It can be a single computer, a cluster of multiple computers, or a cloud server that can flexibly adjust computing resources through cloud technology.

[0043] In a possible implementation, the auxiliary operating system in this embodiment can implement the instruction manual auxiliary display operation based on video interaction only based on the user terminal 11. Specifically, the user terminal 11 displays the instruction manual display request page, and the instruction manual display request page includes an instruction manual display request control. In response to the instruction manual display request control being triggered, the photography component is called to obtain the target video, and image recognition is performed on multiple video frames in the target video respectively. According to the recognition results of the multiple video frames, the product operation requirements are determined, and according to the preset mapping relationship, the target operation scene corresponding to the product operation requirements is determined, and the marking component and / or operation prompt information are determined according to the target operation scene. The marking component is the product component that needs to be introduced at present, and the marking component and / or operation prompt information are highlighted in the photography screen display page. The specific implementation process of each step is detailed in the following embodiment and is not described in detail here.

[0044] In a possible implementation manner, the auxiliary operating system in this embodiment can implement the auxiliary display operation of the instruction manual based on video interaction based on the information interaction among the user terminal 11, the server 12, and the operation terminal 13.

[0045] Optionally, the operation terminal 13 is configured to pre-record video materials in advance to assist the artificial intelligence model in understanding the instruction manual to be converted. The server 12 is configured to obtain the instruction manual to be converted, where the instruction manual is a paper instruction manual and / or a non-interactive video instruction manual, and input the instruction manual to be converted into the artificial intelligence model (i.e., the marking model) to obtain the instruction manual explanation content. The user terminal 11 is configured to display an instruction manual display request page, where the instruction manual display request page includes an instruction manual display request control. In response to the trigger of the instruction manual display request control, the user terminal 11 calls the photography component to obtain the target video, performs image recognition on multiple video frames in the target video respectively, determines the product operation requirements according to the recognition results of the multiple video frames, determines the target operation scenario corresponding to the product operation requirements according to the preset mapping relationship, searches for the marking component and / or operation prompt information corresponding to the target operation scenario from the instruction manual explanation content, where the marking component is the product component that needs to be introduced currently, and highlights the marking component and / or operation prompt information in the photography screen display page. The specific implementation process of each step and the interaction process among devices are described in detail in the following embodiments and will not be elaborated here.

[0046] Optionally, the server 12 is configured to set multiple required operation scenarios, where the operation scenarios can be divided into multiple levels, and then encapsulate the divided operation scenarios into the corresponding video shooting instructions and send the video shooting instructions to the operation terminal 13. The operation terminal 13 is configured to receive the video shooting instructions, display the operation scenarios carried in the video shooting instructions to prompt the staff to shoot the corresponding operation videos. The operation terminal 13 obtains the operation videos shot according to the operation scenarios based on human-computer interaction and sends the operation videos to the server 12. After receiving the operation videos, the server 12 performs component annotation on the products in the operation videos, and then determines whether the components meet the component quantity requirements. If they meet, the server 12 uses each component and the video frame to which the component belongs to train the marking model respectively, so that the marking model can quickly identify the components and the operation scenarios corresponding to the components and determine the marking components that need to be operated by the user currently. The user terminal 11 is configured to request the server 12 to download the marking model, display an instruction manual display request page, where the instruction manual display request page includes an instruction manual display request control. In response to the trigger of the instruction manual display request control, the user terminal 11 calls the photography component to obtain the target video, and then inputs the target video into the marking model, so that the marking model determines the corresponding marking component and / or operation prompt information through image recognition technology, and highlights the marking component and / or the operation prompt information in the photography screen display page.

[0047] Optionally, the information interaction process between the server 12 and the operation terminal 13 can also be as follows: The operation terminal 13 pre - shoots and records an operation video (i.e., video material), and then sends the operation video to the server 12. The server 12 divides the received operation video into operation scenarios. Among them, the operation scenarios can be divided at multiple levels.

[0048] That is to say, the execution order of the server 12 dividing or setting the operation scenario and the operation terminal 13 shooting the operation video can be determined according to the actual situation.

[0049] The system of this embodiment realizes the process of disassembling the user operation into the smallest - granularity instruction - understanding process, thereby realizing the step - by - step disassembly and guiding the user operation, and reducing the user's understanding and operation difficulty.

[0050] The following takes the device type of the user terminal 11 as a mobile phone as an example for illustration.

[0051] Figure 2 is the flowchart of the auxiliary operation method based on video interaction according to the embodiment of the present invention. As Figure 2 shown, the auxiliary operation method based on video interaction includes the following steps:

[0052] Step S201, display a manual display request page, and the manual display request page includes a manual display request control.

[0053] Among them, the manual display request page is a page for receiving the user's request to view the manual. When the user expects to view the manual, the user can click on the application program (or mini - program) that runs the auxiliary operation platform based on video interaction on the display page of the user terminal 11 to enter the application program and display its main page. The main page is the manual display request page, or the main page contains an entrance to the manual display request page, and the manual display request page can be entered through the main page.

[0054] Optionally, the manual display request page includes a product identification input control and an operation category input control.

[0055] Figure 3 is the schematic diagram of the manual display request page according to the embodiment of the present invention. As Figure 3 shown, the manual display request page 300 includes a product identification input control 301, an operation category input control 302, a component identification input control 303, a product component display control 304, and a manual display request control 305.

[0056] Among them, the product identification input control 301 is used to obtain the product identification input by the user. The product identification is used to identify the product type. For example, if the product is a vehicle, the product identification can be the vehicle model; if the product is a mobile phone, the product identification is the mobile phone model.

[0057] The operation category input control 302 is used to obtain the operation category input by the user. The operation category is used to identify the category of the operation that the user expects to perform, and the operation category may include installation, repair, daily maintenance, in-depth maintenance, etc.

[0058] The component identification input control 303 is used to obtain the component identification input by the user. The component identification is used by the user to identify each component in the product.

[0059] The product component display control 304 is used to display the product and each component in the product.

[0060] The above-mentioned component identification input control 303 and product component display control 304 are both associated with the product identification input control 301. When the product identification input control receives the product identification, the product component display control 304 displays the corresponding product, and the selectable options of the component identification input control 303 are the components corresponding to the product. Further, the component identification input control 303 and the product component display control 304 are also associated. When the component identification received by the component identification input control is Component 1, Component 1 in the product component display control 304 is displayed as selected, or when Component 1 in the product component display control 304 is selected, the component identification input control 303 correspondingly displays the product identification of Component 1.

[0061] The instruction manual display request control 305 is used to determine the instruction manual identification and call the photography component to obtain the target video according to the information obtained by the product identification input control 301, operation category input control 302, component identification input control 303, and product component display control 304 after being triggered.

[0062] Among them, the instruction manual identification is used to identify the corresponding instruction manual. There may be multiple instruction manuals that can be displayed in the application program (or applet) running the video interaction-based auxiliary operation platform. The instruction manuals may be different if the products are different, the product components are different, and / or the operation types for the products are different. The product identification input control 301, operation category input control 302, component identification input control 303, and / or product component display control 304 on the instruction manual display request page 300 are all used to obtain the factors for searching the instruction manual. That is to say, the above controls are all used to narrow down the search range of the instruction manual. The smaller the search range, the fewer the corresponding instruction manuals, and then when determining the product operation requirements, the fewer the instruction manuals to be traversed, and the faster the processing speed.

[0063] It should be understood that the controls on the above instruction manual display request page are only examples. In actual use, any data that can narrow down the search range of the instruction manual can be obtained by setting the corresponding controls on the instruction manual display request page. The embodiments of the present invention do not limit the types and quantities of the controls on the instruction manual display request page.

[0064] Step S202, in response to the triggering of the instruction display request control, call the photography component to obtain a target video.

[0065] Taking a mobile phone as an example, the photography component in the mobile phone may include hardware components such as a camera, a flash, a light sensor, and an image processor, as well as software components such as a camera application, a storage component, and an image processing algorithm. The components called during mobile phone photography are a complex system that includes the collaborative work of multiple hardware and software components.

[0066] In a possible implementation manner, in response to the triggering of the instruction display request control, determine the target product according to the product identifier obtained by the product identifier input control, determine the instruction category according to the operation category identifier obtained by the operation category input control, and / or determine the target component according to the component identifier obtained by the component identifier input control 303 or the product component display control 304. Then, determine the instruction identifier according to the target product, the instruction category, and / or the target component, and then call the photography component to obtain a target video containing the target product.

[0067] It should be noted that there may be multiple instruction identifiers determined according to the target product, the instruction category, and / or the target component.

[0068] Optionally, before calling the photography component to obtain the target video, a photography prompt message may also be displayed to prompt the user to align the product and / or component corresponding to the instruction that the user expects to view. For example, the photography prompt message may be "Please align the YY component of the XX product", etc. The target video is a video obtained by the terminal and contains at least one product. The product refers to the product supported by the auxiliary operation platform based on video interaction in this embodiment. Simply put, it is the product for which the auxiliary operation platform based on video interaction can display its relevant instructions.

[0069] In a possible implementation manner, after obtaining the target video, jump from the instruction display request page to the photography screen display page, and the photography screen display page is used to display the photography screen captured by the photography component. That is to say, the display page of the user terminal 11 starts as the instruction display request page first. When the instruction display request control is triggered, it jumps to the photography screen display page until the instruction display ends.

[0070] Step S203, perform image recognition on multiple video frames in the target video respectively.

[0071] Among them, the video frames of the image and video in this step are the video frames in the target video whose pictures contain the product. The image recognition process may include image preprocessing, feature extraction, target detection and recognition, etc. The image recognition method in this embodiment may be an existing image recognition algorithm or an image recognition algorithm that will appear in the future. This embodiment does not limit the image recognition algorithm.

[0072] Step S204, determine the product operation requirements according to the recognition results of the multiple video frames.

[0073] Among them, the product operation requirements may include the target product and the operation requirements.

[0074] In a possible implementation manner, if the target product is not determined before image recognition, the method for determining the target product is to determine the product with the highest frequency of occurrence in the multiple video frames as the target product according to the recognition results corresponding to the multiple video frames.

[0075] In a possible implementation manner, if the target product has been determined before image recognition, for example, the product identifier is obtained based on the above product identifier input control 301, then the product corresponding to the product identifier is also the target product. The method for determining the target product is to perform feature extraction and feature matching on each video frame respectively, and determine the video frames containing the target product according to the matching degree.

[0076] The method for determining the operation requirements is as follows: sort the multiple video frames containing the target product in time sequence, then analyze the user actions in the multiple video frames, and determine the corresponding operation requirements according to the analysis results and the target product. The operation requirements may be the operation categories corresponding to the above instruction manual categories, such as installation, maintenance or repair, etc. Or, the operation requirements may also be specific operations such as power on, power off, installing a program, etc.

[0077] In a possible implementation manner, the user actions that need to be analyzed are actions related to the target product.

[0078] It should be noted that if in the previous step, it is determined that the user expects to perform instruction manual display assistance for a certain component of the product. For example, the component identifier is obtained based on the instruction manual display request page 300, then the product operation requirements in this step may include the target component and the operation requirements.

[0079] Step S205, determine the target operation scenario corresponding to the product operation requirements according to the preset mapping relationship.

[0080] Among them, the preset mapping relationship refers to the mapping relationship between the product operation requirements and the corresponding target operation scenarios set in advance.

[0081] In a possible implementation, the method for determining the preset mapping relationship is as follows: Obtain the instruction manual to be converted, where the instruction manual to be converted is a paper instruction manual and / or a non-interactive video instruction manual. Then, input the instruction manual to be converted into an Artificial Intelligence Model (AI model) to obtain the content of the instruction manual explanation. Then, determine the preset mapping relationship between the product operation requirements and the corresponding target operation scenarios based on the content of the instruction manual explanation.

[0082] That is to say, it is possible to first obtain the instruction manual to be converted, and then input the instruction manual to be converted into the AI model so that the AI model can understand and analyze the content of the traditional paper instruction manual and / or the non-interactive video instruction. Then, take the understanding result of the AI model as the content of the instruction manual explanation. Then, according to the content of the instruction manual explanation and in accordance with the operation process, disassemble the user operation behavior into the smallest granularity process according to the operation process, determine the product operation requirements and operation scenarios corresponding to the smallest granularity process respectively, and determine the preset mapping relationship between the product operation requirements and the operation scenarios with the smallest granularity process as the constraint.

[0083] Optionally, the content of the instruction manual explanation and / or the preset mapping relationship can be downloaded and built into the application for subsequent call at any time.

[0084] Optionally, the content of the instruction manual explanation is relatively detailed, and it includes operation prompt information and / or marked components corresponding to each operation step. Therefore, the marked components and / or operation prompt information corresponding to the target operation scenario can be directly found from the content of the instruction manual explanation.

[0085] In summary, what information is obtained from the content of the instruction manual explanation, or rather, whether the marked components and / or operation prompt information can be directly obtained from the content of the instruction manual explanation depends on the understanding ability of the AI model, that is, the information detail level of the content of the instruction manual explanation.

[0086] Taking the product as a mobile phone as an example to explain steps S204 - S205.

[0087] In the target video, a mobile phone appears in multiple consecutive video frames, and the user's action is to look for the power-on button. Therefore, by analyzing the user's operation behavior, it can be determined that the corresponding product operation requirement is to power on the XX product. Then, with the preset mapping relationship, it is determined that the target operation scenario corresponding to the product operation requirement of powering on the XX product is to indicate the power-on button to the user to meet the power-on requirement.

[0088] Step S206, determine the marked components and / or operation prompt information according to the target operation scenario.

[0089] Among them, the marking component is the product component to be introduced currently. The operation prompt information refers to the prompt information for guiding the user's operation. For example, if the identified component is a screw, the corresponding operation prompt information is "Please screw the screw clockwise."

[0090] In a possible implementation manner, the marking component is also the above-mentioned target component. If the target component has been determined in advance, there is no need to determine the marking component according to the target operation scenario. If the target component has not been determined in advance, it is necessary to determine the marking component according to the target operation scenario. For example, if the target operation scenario is to turn on the target product, it is determined that the marking component should be the power-on button.

[0091] Step S207, prominently display the marking component and / or the operation prompt information on the photography screen display page.

[0092] Specifically, the terminal continuously displays the photography screen. In response to determining the marking component and / or the operation prompt information, the marking component and / or the operation prompt information are prominently displayed in a predetermined form on the photography screen.

[0093] Among them, the predetermined form can be element masking, element magnification, color highlighting, virtual-real highlighting, or using components such as guide lines and shape frames to highlight the marking component and / or the operation prompt information. The technical basis for prominently displaying the marking component in a predetermined form is image recognition technology. That is to say, image recognition technology can be used to identify the component and / or product to be prominently displayed, and then the identified component in the screen is rendered in real time through a preset form. Or, the predetermined form can also be to display symbols such as arrows next to the marking component and / or the operation prompt information that can attract the user's attention. Further, using arrows can also prompt the user's operation direction.

[0094] Figure 4 It is a schematic diagram of the photography screen display page of the embodiment of the present invention. As Figure 4 shown, in the photography screen display page 400, the marking component, that is, component 1, is prominently marked with a marking frame, and the corresponding operation prompt information "Please double-click component 1" is displayed.

[0095] In a possible implementation manner, when displaying the photography screen, in order to improve the screen display effect, interference light elimination, anti-shake, wide-angle camera switching, angle adjustment, and / or flash adjustment and other processes can also be performed.

[0096] In a possible implementation manner, the above steps S203 - step S206 can also be replaced by using a pre-trained marking model to determine the marking component.

[0097] Figure 5 It is a flowchart of the method for using the marking model of the embodiment of the present invention. As Figure 5As shown, the method for using the marking model includes the following steps:

[0098] Step S501, determine the corresponding marking model according to the target product and / or the specification category.

[0099] Specifically, one marking model can correspond to multiple specification categories of multiple products. That is to say, all the specifications in the auxiliary operation platform application or applet based on video interaction can output corresponding auxiliary operations by one or more models. The number of models depends on the model data processing volume and the processing concurrency scale of the program.

[0100] Step S502, obtain the marking model.

[0101] The obtaining method can be to request a download from server 12. The marking model can also be downloaded in advance and saved in the application for direct invocation when needed.

[0102] Step S503, input the target video into the marking model so that the marking model determines the corresponding marking components and / or operation prompt information through image recognition technology.

[0103] That is to say, the marking model can complete image recognition, scene matching, and the determination of marking components and / or operation prompt information.

[0104] The method of the embodiment of the present invention includes displaying a specification display request page, in response to the triggering of the specification display request control in the specification display request page, calling the photography component to obtain the target video, respectively performing image recognition on multiple video frames in the target video, determining the product operation requirements according to the recognition results of the multiple video frames, determining the target operation scenario corresponding to the product operation requirements according to the preset mapping relationship, determining the marking components and / or operation prompt information according to the target operation scenario, where the marking components are the product components that need to be introduced currently, and highlighting the marking components and / or operation prompt information on the photography screen display page. This embodiment analyzes the video captured by the user terminal through video analysis, determines the target operation scenario that matches the user operation behavior in the video, and then matches the corresponding marking components and / or operation prompts for it. The above method realizes disassembling the user operation into the smallest granularity of the specification understanding process, using the device camera to capture the picture in real time, and the large model to analyze and compare the specification content in real time to guide the user's current step operation, so as to realize disassembling and guiding the user operation step by step, reducing the user's understanding and operation difficulty. It solves the one-way information output of traditional paper or non-interactive video specifications, as well as the cognitive deviation between the user's reading of the specification and the actual operation, and reduces the user's understanding threshold and operation difficulty of the specification.

[0105] Figure 6 It is a schematic diagram of the auxiliary operation device based on video interaction according to the embodiment of the present invention. AsFigure 6 As shown in the figure, the video interaction-based auxiliary operation device includes:

[0106] A first display module 601 for displaying a manual display request page, where the manual display request page includes a manual display request control.

[0107] A component calling module 602 for, in response to the manual display request control being triggered, calling a photography component to obtain a target video.

[0108] An image recognition model 603 for respectively performing image recognition on multiple video frames in the target video.

[0109] A first determination module 604 for determining product operation requirements according to the recognition results of the multiple video frames.

[0110] A second determination module 605 for determining a target operation scenario corresponding to the product operation requirements according to a preset mapping relationship.

[0111] A third determination module 606 for determining a marking component and / or operation prompt information according to the target operation scenario, where the marking component is a product component that needs to be introduced currently.

[0112] A second display module 607 for highlighting the marking component and / or the operation prompt information in a photography screen display page.

[0113] The device of this embodiment is used to display a manual display request page, in response to the manual display request control in the manual display request page being triggered, call a photography component to obtain a target video, respectively perform image recognition on multiple video frames in the target video, determine product operation requirements according to the recognition results of the multiple video frames, determine a target operation scenario corresponding to the product operation requirements according to a preset mapping relationship, determine a marking component and / or operation prompt information according to the target operation scenario, where the marking component is a product component that needs to be introduced currently, and highlight the marking component and / or operation prompt information in a photography screen display page. The device of this embodiment analyzes the video screen collected by the user terminal to determine the target operation scenario that matches the user operation behavior in the video, and then matches the corresponding marking component and / or operation prompt for it. The above device realizes disassembling the user operation into the smallest granularity of the manual understanding process, thereby realizing step-by-step disassembling and guiding the user operation, and reducing the user's understanding and operation difficulty.

[0114] Figure 7 It is a schematic diagram of an electronic device according to an embodiment of the present invention. As Figure 7 shown, Figure 7The electronic device shown is an auxiliary operation device based on video interaction, which includes a general computer hardware structure, and at least includes a processor 701 and a memory 702. The processor 701 and the memory 702 are connected through a bus 703. The memory 702 is adapted to store instructions or programs executable by the processor 701. The processor 701 can be an independent microprocessor or a set of one or more microprocessors. Thus, by executing the instructions stored in the memory 702, the processor 701 executes the method flow of the embodiment of the present invention as described above to implement data processing and control of other devices. The bus 703 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to a display controller 704, a display device, and an input / output (I / O) device 705. The input / output (I / O) device 705 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a body-sensing input device, a printer, and other devices well-known in the art. Typically, the input / output device 705 is connected to the system through an input / output (I / O) controller 706.

[0115] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a device (equipment), or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be implemented as a computer program product on one or more computer-readable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0116] The present application is described with reference to the flowcharts of methods, devices (equipment), and computer program products according to the embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0117] These computer program instructions can be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device, and the instruction device implements the process Figure 1 specified functions in one process or multiple processes.

[0118] These computer program instructions can also be provided to the processor of a general computer, a special computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device for implementing the Figure 1 specified functions in one process or multiple processes.

[0119] The technical solution of the embodiment of the present invention is to display the instruction manual display request page. In response to the triggering of the instruction manual display request control in the instruction manual display request page, the photography component is called to obtain the target video. Image recognition is performed on multiple video frames in the target video respectively. According to the recognition results of the multiple video frames, the product operation requirements are determined. According to the preset mapping relationship, the target operation scenario corresponding to the product operation requirements is determined. According to the target operation scenario, the marking component and / or the operation prompt information are determined. The marking component is the product component that needs to be introduced currently. The marking component and / or the operation prompt information are prominently displayed in the photography screen display page. In this embodiment, through video analysis of the video screen collected by the user terminal, the target operation scenario matching the user operation behavior in the video is determined, and then the corresponding marking component and / or operation prompt are matched for it. The above technical solution realizes disassembling the user operation into the smallest granularity of the instruction manual understanding process, so as to realize guiding the user operation step by step and reducing the user's understanding and operation difficulty.

[0120] Another embodiment of the present invention relates to a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned partial or all method embodiments are realized.

[0121] Another embodiment of the present invention relates to a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the above-mentioned partial or all method embodiments are realized.

[0122] That is, those skilled in the art can understand that all or part of the steps of implementing the above-mentioned embodiment methods can be completed by specifying relevant hardware through a program. This program is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0123] The foregoing are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. An auxiliary operation method based on video interaction, characterized in that: The method comprises: Displaying a manual display request page, wherein the manual display request page includes a manual display request control; In response to the manual display request control being triggered, calling the photography component to obtain the target video; Performing image recognition on multiple video frames in the target video respectively; Determining a product operation requirement based on the recognition results of the multiple video frames; Determine the target operation scenario corresponding to the product operation requirement according to the preset mapping relationship; Determine a marked component and / or operation prompt information according to the target operation scenario, wherein the marked component is a product component that currently needs to be introduced; The marking component and / or the operation prompt information are highlighted in the photography screen display page.

2. The method according to claim 1, characterized in that The manual display request page also includes a product identification input control and an operation category input control; In response to the instruction manual display request control being triggered, calling the photography component to obtain the target video includes: In response to the manual display request control being triggered, determining a target product according to the product identifier obtained by the product identifier input control, and / or determining a manual category according to the operation category identifier obtained by the operation category input control; The photography component is called to obtain a target video containing the target product.

3. The method according to claim 2, characterized in that The method further comprises: Determine a corresponding marking model according to the target product and / or the instruction manual category; The target video is input into the marking model so that the marking model determines the corresponding marking component and / or operation prompt information through image recognition technology.

4. The method according to claim 1, characterized in that: The product operation requirements include target products and operation requirements; The determining of the product operation requirement according to the recognition results of the multiple video frames includes: According to the recognition results corresponding to the multiple video frames, the product with the highest appearance frequency is determined as the target product; Analyzing the user actions in the plurality of video frames in a time sequence; According to the analysis results and the target product, the corresponding operation requirements are determined.

5. The method according to claim 1, characterized in that The method further comprises: Displaying the photographic image display page, wherein the photographic image display page is used to display the photographic image captured by the photographic component; The highlighting of the marking component and / or the operation prompt information in the photographing image display page includes: In response to determining the marking component and / or the operation prompt information, the marking component and / or the operation prompt information are highlighted in the photographic image in the form of an element mask.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Obtaining the instructions to be converted, where the instructions to be converted are paper instructions and / or non-interactive video instructions; Inputting the instruction manual to be converted into an artificial intelligence model to obtain the explanation content of the instruction manual; Search the marking components and / or operation prompt information corresponding to the target operation scenario from the explanation content of the instruction manual.

7. An auxiliary operating system based on video interaction, characterized in that: The system comprises: The server is configured to obtain a manual to be converted, wherein the manual is a paper manual and / or a non-interactive video manual, and input the manual to be converted into an artificial intelligence model to obtain an explanation content of the manual; The user end is configured to display a manual display request page, which includes a manual display request control. In response to the manual display request control being triggered, the photography component is called to obtain a target video, and image recognition is performed on multiple video frames in the target video respectively. According to the recognition results of the multiple video frames, the product operation requirements are determined, and according to a preset mapping relationship, the target operation scene corresponding to the product operation requirements is determined, and the marking component and / or operation prompt information corresponding to the target operation scene are searched from the explanation content of the manual. The marking component is the product component that currently needs to be introduced, and the marking component and / or the operation prompt information are highlighted in the photography screen display page.

8. An auxiliary operation device based on video interaction, characterized in that: The device comprises: A first display module, configured to display a manual display request page, wherein the manual display request page includes a manual display request control; A component calling module, configured to call a photography component to obtain a target video in response to the instruction manual display request control being triggered; An image recognition model, used to perform image recognition on multiple video frames in the target video respectively; A first determination module, configured to determine a product operation requirement according to the recognition results of the plurality of video frames; A second determination module is used to determine the target operation scenario corresponding to the product operation requirement according to a preset mapping relationship; A third determination module is used to determine a marking component and / or operation prompt information according to the target operation scenario, wherein the marking component is a product component that needs to be introduced currently; The second display module is used to highlight the marking component and / or the operation prompt information in the photography picture display page.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images