Method and device for generating guidance video for equipment use, electronic equipment and storage medium

By generating device usage guidance videos, it solves the problem that users find it difficult to use unfamiliar devices, provides tutorials suitable for users' operations, and improves user experience and fitness effects.

CN118949383APending Publication Date: 2024-11-15SAMSUNG ELECTRONICS CHINA R&D CENT +1
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411020127.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

When facing unfamiliar devices, it is difficult for users to understand how to use them correctly, resulting in a reduced user experience and a failure to achieve fitness goals.

Method used

By generating intent prompt words based on user intent information, the image of the target device is extracted from the scene image of the target area, the intent prompt words and the image of the target device generate guidance video, and the equipment is constructed using guidance video generation model to provide suitable Tutorial on the use of the device operated by the user.

Benefits of technology

Effectively guide users to use the equipment correctly, improve their interest and experience in using the equipment, ensure that users can exercise reasonably and avoid incorrect movements causing damage to their bodies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118949383A_ABST
    Figure CN118949383A_ABST
Patent Text Reader

Abstract

The invention provides a guidance video generation method and device for equipment use, electronic equipment and a storage medium, and relates to the field of computer vision algorithms, in particular to the technical field of equipment application guidance. According to the specific implementation scheme, an intention prompt word is generated based on intention information of a user; extracting an image of the target equipment from the scene image of the target area; and generating a guidance video for using the target equipment based on the intention prompt word and the image of the target equipment. The correct use course suitable for the equipment operated by the user is generated, and the user is effectively guided to use the equipment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision algorithms, specifically to the field of device application guidance technology, and in particular to a method, device, electronic device and storage medium for generating a guidance video for device use. Background Art

[0002] With the continuous advancement of technology, equipment manufacturing has become more intelligent and diversified. When users are faced with some unfamiliar equipment, they may not understand how to choose and use it in time, which will reduce their interest in using the equipment and their experience. For example, in a gym, there may be a large number of fitness equipment to choose from. If users use unfamiliar fitness equipment and exercise in the wrong way, they will not be able to achieve their fitness goals. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device and storage medium for generating a guide video for device use, which generates a correct use tutorial of the device suitable for user operation and effectively guides the user to use the device.

[0004] According to a first aspect of the present disclosure, a method for generating an instructional video for device use is provided, including: generating intention prompt words based on user intention information; extracting an image of a target device from a scene image of a target area; and generating an instructional video for using the target device based on the intention prompt words and the image of the target device.

[0005] In some embodiments, the above-mentioned intention information also includes device information; the above-mentioned extracting the image of the target device from the scene image of the target area includes: extracting images of multiple devices from the scene image; based on the device information, determining the image of the target device from the images of multiple devices.

[0006] In some embodiments, the above-mentioned extracting the image of the target device from the scene image of the target area includes: extracting images of multiple devices from the scene image; determining the application field corresponding to the intention information; and based on the application field, determining the image of the target device from the images of multiple devices.

[0007] In some embodiments, the above-mentioned generation of an instruction video for using the target device based on the intention prompt words and the image of the target device includes: inputting the intention prompt words and the image of the target device into a device usage instruction video generation model of the application field corresponding to the intention information to generate an instruction video for the target device.

[0008] In some embodiments, the construction process of the above-mentioned device usage guidance video generation model includes: constructing a first data set using device images, intention prompt words and device usage guidance videos of devices in the application field as samples; using the first data set to fine-tune the latent diffusion model to obtain the device usage guidance video generation model; wherein the latent diffusion model is a diffusion model that can handle text-generated image tasks and image-generated image tasks.

[0009] In some embodiments, the method further includes: in response to there being multiple target devices, sorting the instruction videos for the multiple target devices based on the attribute information of the user; and pushing a first preset number of instruction videos that are sorted first to the user.

[0010] In some embodiments, the above-mentioned sorting of the guidance videos of multiple target devices based on the user's attribute information includes: determining the matching degree between the attribute information and the applicable standards of the multiple target devices; and sorting the guidance videos of the multiple target devices in order of matching degree from high to low.

[0011] In some embodiments, the generating of the intent prompt word based on the intent information includes: inputting the intent information into a prompt word generation model to determine the intent prompt word.

[0012] According to a second aspect of the present disclosure, a device for generating an instructional video for device use is provided, comprising: a first generation module, configured to generate intention prompt words based on user intention information; an extraction module, configured to extract an image of a target device from a scene image of a target area; and a second generation module, configured to generate an instructional video for using the target device based on the intention prompt words and the image of the target device.

[0013] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the guidance video generation method for device use provided by the first aspect.

[0014] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method for generating a guidance video for device use provided in the first aspect.

[0015] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method for generating a guidance video for device use provided in the first aspect is implemented.

[0016] The method, apparatus, electronic device and storage medium for generating a guidance video for device use provided by the present disclosure generate intention prompt words based on user intention information; extract an image of a target device from a scene image of a target area; generate a guidance video for using the target device based on the intention prompt words and the image of the target device; and generate a correct usage tutorial for the device suitable for user operation by using the device and user intention identified from the current environment, thereby effectively guiding the user to use the device.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0019] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0020] Figure 2 is a flow chart of an embodiment of a method for generating a guidance video for device use according to the present disclosure;

[0021] Figure 3 is a schematic diagram 300 of a potential diffusion model according to the present disclosure;

[0022] Figure 4 is a flowchart of another embodiment of a method for generating a guidance video for device use according to the present disclosure;

[0023] Figure 5 is a flow chart of another embodiment of a method for generating a guidance video for device use according to the present disclosure;

[0024] Figure 6 is a structural schematic diagram of an embodiment of a device for generating a guidance video for device use according to the present disclosure;

[0025] Figure 7 The block diagram of an electronic device used to implement the method for generating a guidance video for device use according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the method for generating a guidance video for device usage and the apparatus for generating a guidance video for device usage of the present disclosure can be applied.

[0028] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0029] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104, or can interact with each other through server 105. Terminal devices 101, 102, 103 can be installed with various client applications, such as camera, video chat, video conferencing applications, etc.

[0030] The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the terminal devices 101, 102, 103 are software, they may be installed in the above electronic devices. They may be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0031] The server 105 can provide various services. For example, the server 105 analyzes and processes the user's intention information and the scene image of the target area obtained from the terminal devices 101, 102, and 103 to obtain a guidance video for using the target device.

[0032] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0033] It should be noted that the method for generating a guidance video for device use provided in the embodiments of the present disclosure is generally executed by the server 105 , and accordingly, the device for generating a guidance video for device use is generally disposed in the server 105 .

[0034] It should be understood that Figure 1 The number of mobile devices, networks and servers in the embodiment is only for illustration. Any number of mobile devices, networks and servers may be provided as required.

[0035] Continue to refer Figure 2 , which shows a process 200 of an embodiment of a method for generating a guidance video for device use according to the present disclosure. The guidance video generation method comprises the following steps:

[0036] Step 201: Generate intention prompt words based on user intention information.

[0037] In this embodiment, the execution subject (eg, Figure 1 The server 105 shown in the figure may first obtain the information of the user through the terminal device (eg Figure 1 The terminal devices 101, 102, 103 shown in the figure) input the intention information, and then generate the intention prompt words based on the intention information.

[0038] In some optional implementations of this embodiment, the above-mentioned intention information includes task content and task objectives, wherein the task content is used to represent the specific description of the task to be performed by the user, and the task objective represents the effect that the user wants to achieve by performing the task.

[0039] As an example, the above-mentioned execution entity may first obtain information input by the user through touch, text and / or voice, and then perform semantic recognition on the information to determine the user's intention information.

[0040] As another example, the above-mentioned execution entity may provide a preset template to the user, and receive the preset template filled in by the user to determine the user's intention information, wherein the preset template includes at least two filling items: task content and task goal.

[0041] In some optional implementations of this embodiment, the generating of the intent prompt word based on the intent information includes: inputting the intent information into a prompt word generation model to determine the intent prompt word.

[0042] Among them, the prompt word generation model can extract keywords related to the topic or intent based on the understanding of the sentence. These keywords can be nouns, verbs, adjectives, etc., which can summarize the core content and key points of the sentence. In short, the prompt word generation model is used to condense long sentences into more concise and refined expressions. For example, if the intent information "I want to exercise my abdominal core muscles" is input, the intent prompt word "exercise, abdominal muscles, core muscles" is generated.

[0043] As an example, the prompt word generation model may be a Transformer model (a deep learning model based on a self-attention mechanism), and its training process may include:

[0044] Step 1, collect training sample data, where training samples are represented by text-keyword pairs, for example: "I want to use dumbbells to practice squats to increase muscle and improve core strength..." - "dumbbells, squats, muscle growth, quarter squats..."; for example, "I also want to use dumbbells to practice squats to achieve bodybuilding effects..." - "dumbbells, squats, bodybuilding, powerlifting, increase leg circumference..."

[0045] Step 2: Perform data preprocessing on the text in the training sample to obtain the word sequence corresponding to the text, wherein the data preprocessing includes cleaning and removing useless punctuation and special symbols, dividing the continuous text sequence into independent words and deduplicating the text, etc.

[0046] Step 3: According to word embedding technology (such as word2vec, etc.), the word sequence is mapped to a high-dimensional space to obtain a word sequence encoding vector;

[0047] Step 4: Take the word sequence encoding vector as input and the keyword as output to train the prompt word generation model.

[0048] Step 202: extracting an image of a target device from a scene image of a target area.

[0049] In this embodiment, the execution subject may obtain a scene image of the target area captured by the user through the terminal device, and then extract the image of the target device from the scene image.

[0050] In some optional implementations of this embodiment, the target area refers to the user's current activity area, such as a gym, a kitchen, etc.

[0051] The above-mentioned extracting the image of the target device from the scene image of the target area includes: extracting images of multiple devices from the scene image; and determining the image of the target device from the images of the multiple devices.

[0052] The extracting of images of multiple devices from the scene image includes: using a target detection algorithm to obtain device images of all devices in the scene image.

[0053] As an example, the above-mentioned execution entity can input the scene image into the target detection model to obtain the device images of all devices in the scene image.

[0054] Among them, the above-mentioned target detection model can be constructed using algorithms such as R-CNN (Region-CNN, regional convolutional neural network) series, YOLO (You Only Look Once, object detection) series, SSD (Single Shot MultiBox Detector, single-stage multi-layer target detection) and CenterNet (target detection based on the center point of the object). The training of these algorithms is generally to learn the features of the input image through the network, identify the target in the image by analyzing the features, and output the detection box and its category corresponding to the target.

[0055] The above-mentioned determining the image of the target device from the images of the plurality of devices includes: determining the image of the target device from the images of the plurality of devices based on the task content.

[0056] As an example, the execution subject may select one or more images matching the task content from images of multiple devices as the image of the target device.

[0057] Step 203: Generate a guidance video for using the target device based on the intention prompt word and the image of the target device.

[0058] In this embodiment, after determining the intention prompt word and the image of the target device, the execution subject may generate a guidance video for using the target device based on the intention prompt word and the image of the target device.

[0059] In some optional implementations of the present embodiment, the above-mentioned generation of an instruction video for using the target device based on the intention prompt word and the image of the target device includes: determining whether there is a prompt word in a pre-established prompt word library whose matching degree with the intention prompt word is higher than a threshold; if so, searching for an instruction video that matches both the prompt word and the image of the target device from a pre-established instruction video library, and determining the instruction video as an instruction video for using the target device, wherein each instruction video pre-stored in the instruction video library uniquely corresponds to a prompt word and a device image.

[0060] In some optional implementations of the present embodiment, the above-mentioned generation of an instruction video for using the target device based on the intention prompt words and the image of the target device includes: inputting the intention prompt words and the image of the target device into a device usage instruction video generation model for the application field corresponding to the intention information, and generating an instruction video for the target device.

[0061] The application field corresponding to the above-mentioned intent information refers to the task field in which the task content in the intent information is located, for example, fitness intent corresponds to the fitness field, cooking intent corresponds to the cooking field, and so on.

[0062] The construction process of the above-mentioned device usage guidance video generation model includes: constructing a first data set using device images, intention prompt words and device usage guidance videos of devices in the application field as samples; using the first data set to fine-tune the potential diffusion model to obtain the device usage guidance video generation model.

[0063] Among them, the above-mentioned fine-tuning of the latent diffusion model using the first data set to obtain a device usage guidance video generation model includes: inputting the device image and intention prompt words in the sample into the latent diffusion model to obtain a video output by the latent diffusion model; calculating the loss between the video and the device usage guidance video in the sample; using the loss to optimize the latent diffusion model; repeating the above steps until the preset conditions are met.

[0064] The above preset condition may be that the loss between the video output by the latent diffusion model and its true value is less than a preset threshold, or the training deployment reaches a preset number of steps.

[0065] The above calculation of the loss between the video and the device usage guidance video in the sample includes: respectively sending the video and the device usage guidance video in the sample into an image encoder for parsing into image sequences, so that the image encoder maps the image sequence from pixel space to feature space to obtain a feature sequence, comparing the similarity between the feature sequence of the video and the feature sequence of the device usage guidance video in the sample, and determining the loss between the video and the device usage guidance video in the sample based on the similarity.

[0066] Continue to refer Figure 3 , which shows a schematic diagram 300 of the structure of a potential diffusion model according to the present disclosure. The potential diffusion model includes an image encoder, a text encoder, a diffusion model and an image and video decoder.

[0067] Among them, the latent diffusion model is a model that is trained with a large amount of image-text data and image-image data, and can handle text-to-image tasks and image-to-image tasks. The latent diffusion model inputs the image into the image encoder to encode the image vector, and inputs the text into the text encoder to encode the text vector; then the image vector and text vector are input into the diffusion model to obtain the vector representation of the image or video in the latent space; finally, the image video decoder is used to convert the vector representation of the latent space into an image or video for output.

[0068] For downstream tasks of fine-tuning the latent diffusion model, a Lora model (low-rank adaptive model) is added to the latent diffusion model. In essence, a bypass is added to the original latent diffusion model, which includes a dimensionality reduction module and a dimensionality increase module. It is only necessary to retrain the dimensionality reduction module and the dimensionality increase module based on the original latent diffusion model using the training data of the downstream tasks. On the premise that the model can adapt to downstream tasks, the fine-tuning speed of the model is accelerated, and the generation ability and stability of the model are improved.

[0069] The method for generating a guidance video for device use provided in the embodiment of the present disclosure utilizes the device and user intention identified from the current environment to generate a correct usage tutorial for the device suitable for user operation, thereby effectively guiding the user to use the device. The method can be applied to mobile devices, such as mobile phones, tablets, augmented reality glasses, and other devices, and can provide users with customized device use guidance services regardless of the environment.

[0070] Continue to refer Figure 4 , which shows a process 400 of another embodiment of a method for generating a guidance video for device use according to the present disclosure. The guidance video generation method comprises the following steps:

[0071] Step 401: Generate intention prompt words based on user intention information.

[0072] In this embodiment, the specific operations of step 401, step 402 and step 407 are already described in Figure 2 In the illustrated embodiment, steps 201 to 203 are described in detail and will not be repeated here.

[0073] Step 402: extract images of multiple devices from the scene image.

[0074] Step 403: Determine whether the intent information includes device information.

[0075] In this embodiment, the execution subject (eg, Figure 1 The server 105 shown in the figure may first obtain the information of the user through the terminal device (eg Figure 1 The intention information is input by the terminal devices 101, 102, 103) shown in the figure, and it is determined whether the intention information includes device information.

[0076] In some optional implementations of this embodiment, the above-mentioned device information represents the device that the user expects to use to perform his task.

[0077] As an example, the execution entity may perform semantic recognition on the user's intention information to determine whether the intention information includes device information.

[0078] As another example, the preset template provided by the execution entity to the user may include an optional item for device information, and the execution entity receives the preset template filled in by the user to determine whether the intent information includes device information.

[0079] Step 404: In response to the intention information including device information, an image of a target device is determined from images of a plurality of devices based on the device information.

[0080] In this embodiment, after determining that the intention information includes device information, the execution subject directly uses the device corresponding to the device information as the target device, and determines the image of the target device from the images of multiple devices.

[0081] Step 405: In response to the intent information not including device information, determine the application field corresponding to the intent information.

[0082] In this embodiment, after determining that the intent information does not include device information, the execution subject determines the application field corresponding to the intent information. The application field corresponding to the intent information refers to the task field where the task content in the intent information is located, such as fitness intent corresponding to the fitness field, cooking intent corresponding to the cooking field, etc.

[0083] Step 406: Based on the application field, determine the image of the target device to be extracted from the images of the multiple devices.

[0084] In this embodiment, the execution subject determines the application field corresponding to the intention information, and screens the images of multiple devices based on the application field to obtain the image of the target device.

[0085] In some optional implementations of this embodiment, the above-mentioned screening of images of multiple devices based on the application field to obtain the image of the target device includes:

[0086] Images of devices that do not belong to the application field are filtered out, and images of the remaining multiple devices are used as images of target devices.

[0087] Step 407: Generate a guidance video for using the target device based on the intention prompt word and the image of the target device.

[0088] The method for generating a guidance video for device usage provided by the embodiment of the present disclosure generates a usage guidance video for the device corresponding to the device information when the user provides the device information; and generates and recommends to the user usage guidance videos for all devices in the application field corresponding to the intent information in the user activity space when the user does not provide the device information, thereby improving the user's usage experience.

[0089] Continue to refer Figure 5, which shows a process 500 of another embodiment of a method for generating a guidance video for device use according to the present disclosure. The guidance video generation method includes the following steps:

[0090] Step 501: Generate intention prompt words based on user intention information.

[0091] Step 502: extracting an image of a target device from a scene image of a target area.

[0092] Step 503: Generate a guidance video for using the target device based on the intention prompt word and the image of the target device.

[0093] In this embodiment, the specific operations of step 501, step 502 and step 507 are already described in Figure 2 In the illustrated embodiment, steps 201 to 203 are described in detail and will not be repeated here.

[0094] Step 504: In response to there being multiple target devices, sort the instructional videos for the multiple target devices based on the attribute information of the user.

[0095] In this embodiment, the execution subject (eg, Figure 1 After determining the instruction video for using the target device, the server 105 shown in the figure can determine whether the number of target devices is multiple. In response to the number of target devices being one, the instruction video for using the target device is pushed to the user. In response to the number of target devices being multiple, the instruction videos for the multiple target devices are sorted based on the attribute information of the user.

[0096] In some optional implementations of this embodiment, the above-mentioned attribute information obtains the user's personal information and physical indicators based on the user's authorization. The personal information includes age, gender, height, weight, etc., and the physical indicators include the user's physical condition, etc.

[0097] The above-mentioned sorting of the instruction videos of the multiple target devices based on the attribute information of the user includes: determining the matching degree between the attribute information and the applicable standards of the multiple target devices; sorting the instruction videos of the multiple target devices in descending order of matching degree. The above-mentioned applicable standards indicate the applicable population of the device, for example, for females, 30 to 40 years old, and 160 cm tall.

[0098] Step 505: Push a first preset number of instructional videos that are ranked first to the user.

[0099] The method for generating an instructional video for device use provided by the embodiment of the present disclosure recommends an instructional video based on user attribute information when there are multiple instructional videos that can be recommended, thereby improving the accuracy of instructional video recommendations.

[0100] The method for generating a guidance video for equipment use disclosed in the present invention is used in various fields. Taking the fitness field as an example, the fitness guidance methods in the prior art include: (1) Professional fitness coaches can provide guidance on the use of fitness equipment and provide customized fitness plans, but the price is high. (2) Fitness equipment manufacturers will provide teaching videos and generate links. Users can scan the links to view the usage tutorials, but it is easy to leak personal information and cannot directly show the user's fitness intentions. (3) Users can search for guidance videos for these fitness equipment online, but the search operation is cumbersome and the accuracy is low. The method provided in the present invention identifies the fitness equipment in the current environment, selects suitable fitness equipment for the user based on the user's fitness intention, and generates a customized usage guidance video for the selected fitness equipment to guide the user's fitness, help the user exercise reasonably, and avoid damage to the body caused by incorrect movements.

[0101] Continue to refer Figure 6 , Figure 6 A structural diagram 600 of an embodiment of a device for generating a video guide for device use according to the present disclosure is illustrated; Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0102] like Figure 6 As shown, the apparatus for generating a guidance video for device use in this embodiment may include a first generation module 601, an extraction module 602, and a second generation module 603. The first generation module 601 is configured to generate an intention prompt word based on the user's intention information; the extraction module 602 is configured to extract an image of the target device from the scene image of the target area; and the second generation module 603 is configured to generate a guidance video for using the target device based on the intention prompt word and the image of the target device.

[0103] In this embodiment, the specific processing of the first generation module 601, the extraction module 602 and the second generation module 603 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of steps 201 - 203 in the corresponding embodiment are not repeated here.

[0104] In some optional implementations of this embodiment, the extraction module 602 includes a first extraction unit and a first determination unit, wherein the first extraction unit is configured to extract images of multiple devices from the scene image in response to device information included in the intent information; and the first determination unit is configured to determine the image of the target device from the images of the multiple devices based on the device information.

[0105] In some optional implementations of this embodiment, the extraction module 602 includes a second extraction module, a second determination module, and a third determination module. The second extraction module is configured to extract images of multiple devices from the scene image; the second determination module is configured to determine the application field corresponding to the intention information; and the third determination module is configured to determine the image of the extraction target device from the images of the multiple devices based on the application field.

[0106] In some optional implementations of this embodiment, the second generation module 603 includes a generation unit, wherein the generation unit is configured to generate a guidance video for the target device by inputting the intention prompt word and the image of the target device into a device usage guidance video generation model in the application field corresponding to the intention information.

[0107] In some optional implementations of the present embodiment, the process of constructing the above-mentioned device usage guidance video generation model includes: constructing a first data set using device images, intention prompt words and device usage guidance videos of devices in the application field as samples; using the first data set to fine-tune the latent diffusion model to obtain the device usage guidance video generation model; wherein the latent diffusion model is a diffusion model that can handle text-generated image tasks and image-generated image tasks.

[0108] In some optional implementations of the present embodiment, the above-mentioned device also includes a pushing module, the pushing module includes a sorting unit and a pushing unit, wherein the sorting unit is configured to sort the guidance videos of multiple target devices based on the attribute information of the user in response to the number of target devices being multiple; and the pushing unit is configured to push a first preset number of guidance videos that are sorted in front to the user.

[0109] In some optional implementations of this embodiment, the sorting unit includes a matching degree determination subunit and a sorting subunit, wherein the matching degree determination subunit is configured to determine the matching degree between the attribute information and the applicable standards of multiple target devices; and the sorting subunit is configured to sort the guidance videos of the multiple target devices in order of matching degree from high to low.

[0110] In some optional implementations of this embodiment, the first generation module 601 includes a second determination unit, wherein the second determination unit is configured to input the intention information into the prompt word generation model to determine the intention prompt word.

[0111] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0112] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0113] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0114] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0115] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a method for generating a guidance video for use with a device. For example, in some embodiments, a method for generating a guidance video for use with a device may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the method for generating a guidance video for use with a device described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the method for generating a guidance video for device use in any other appropriate manner (eg, by means of firmware).

[0116] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0118] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0120] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0121] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0122] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0123] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0124] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a video guide for device use, include: Generate intention prompt words based on user's intention information; extracting an image of a target device from a scene image of a target area; Based on the intention prompt word and the image of the target device, a guidance video for using the target device is generated.

2. The method according to claim 1, in, The intent information also includes device information; The step of extracting the image of the target device from the scene image of the target area includes: extracting images of a plurality of devices from the scene image; Based on the device information, an image of the target device is determined from the images of the plurality of devices.

3. The method according to claim 1, in, The step of extracting the image of the target device from the scene image of the target area includes: extracting images of a plurality of devices from the scene image; Determine the application field corresponding to the intention information; Based on the application field, the image of the target device is determined to be extracted from the images of the multiple devices.

4. The method according to claim 1, in, The step of generating a guidance video for using the target device based on the intention prompt word and the image of the target device includes: The intention prompt word and the image of the target device are input into a device usage guidance video generation model in the application field corresponding to the intention information to generate a guidance video for the target device.

5. The method according to claim 4, in, The device uses a construction process of a guidance video generation model, including: Constructing a first data set using device images, intention prompt words, and device usage guidance videos of devices in the application field as samples; Fine-tuning the potential diffusion model using the first data set to obtain the device usage guidance video generation model; The potential diffusion model is a diffusion model that can handle text-generated-graph tasks and graph-generated-graph tasks.

6. The method according to claim 1, in, The method further comprises: In response to the target device being multiple, sorting the instruction videos of the multiple target devices based on the attribute information of the user; The first preset number of instructional videos in the front order are pushed to the user.

7. The method according to claim 6, in, The step of sorting the instruction videos of the plurality of target devices based on the attribute information of the user comprises: determining a degree of match between the attribute information and applicable standards of the plurality of target devices; The instruction videos of the multiple target devices are sorted in descending order of matching degree.

8. The method according to claim 1, wherein the generating of the intention prompt word is based on the intention information. include: The intention information is input into a prompt word generation model to determine the intention prompt word.

9. A device for generating a video guide for use of a device, include: A first generating module is configured to generate an intention prompt word based on the user's intention information; an extraction module configured to extract an image of a target device from a scene image of a target area; The second generating module is configured to generate an instructional video for using the target device based on the intention prompt word and the image of the target device.

10. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Cooking instruction method, apparatus, apparatus and storage medium for AR scene

    CN109299293A

  • Equipment guidance method and device, electronic equipment and storage medium

    CN117424999A

  • System for creating and managing workout menu

    JP2003024491A

  • Augmenting transmitted video data

    US10950049B1

  • System for synchronizing video output based on detected behavior

    US11861944B1