Methods, apparatus, devices, and storage media for interaction

Abstract

Solves the problems of conventional technology. [Solution] Embodiments of the present invention provide a method, apparatus, device, and storage medium for interaction. The method includes, in response to receiving an interaction request from a user, collecting interaction content using a content collection device, wherein the interaction content includes at least image content; performing object detection on the image content to obtain object detection results, wherein the object detection results indicate at least one object within the image content; determining, based on the object detection results, a target object classified as a point of interest and interaction requirements for the target object from the at least one object; and performing a target operation associated with the target object based on the interaction requirements.

Description

Technical Field

[0001] Exemplary embodiments of the present invention generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, computer-readable storage media, and computer program products for interaction.

Background Art

[0002] With the development of artificial intelligence technology, various types of product forms have emerged. For example, artificial intelligence products (e.g., digital assistants) perform interactions between humans and computers via voice or text, providing users with a great deal of convenience. However, the performance of conventional digital assistants in image-based human-computer interaction still has room for improvement.

Summary of the Invention

[0003] In a first aspect of the present invention, a method for interaction is provided. The method includes collecting interaction content using a content collection device in response to receiving a user's interaction request, where the interaction content includes at least image content; performing object detection on the image content to obtain an object detection result, where the object detection result indicates at least one object in the image content; determining, based on the object detection result, a target object classified as an interesting point and an interaction requirement for the target object from the at least one object; and performing a target operation associated with the target object based on the interaction requirement.

[0004] In a second aspect of the present invention, an interaction apparatus is provided. The apparatus comprises a collection module configured to collect interaction content using a content collection device in response to receiving an interaction request from a user, wherein the interaction content includes at least image content; an acquisition module configured to perform object detection on the image content to obtain object detection results, wherein the object detection results indicate at least one object within the image content; a determination module configured to determine, based on the object detection results, a target object classified as a point of interest and interaction requirements for the target object from at least one object; and an execution module configured to perform a target operation associated with the target object based on the interaction requirements.

[0005] A third aspect of the present invention provides an electronic device comprising at least one processing unit and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device causes the device to perform the method of the first aspect.

[0006] A fourth embodiment of the present invention provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method of the first embodiment can be realized.

[0007] In a fifth embodiment of the present invention, a computer program product is provided, the computer program product comprising a computer program, wherein the computer program is executed by a processor, thereby realizing the method of the first embodiment of the present invention.

[0008] It should be understood that the information described in the summary of the present invention is not intended to limit the main or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will be readily apparent from the following description. [Brief explanation of the drawing]

[0009] Referring to the following detailed description in conjunction with the drawings will further clarify the above-mentioned features and other features, advantages, and aspects of each embodiment of the present invention. In the drawings, the same or similar reference numerals indicate the same or similar elements.

[0010] [Figure 1] A schematic diagram of an exemplary environment in which embodiments of the present invention can be implemented is shown.

[0011] [Figure 2] A flowchart of the interaction process according to some embodiments of the present invention is shown below.

[0012] [Figure 3A] A schematic diagram of image content according to some embodiments of the present invention is shown. [Figure 3B] A schematic diagram of image content according to some embodiments of the present invention is shown. [Figure 3C] A schematic diagram of image content according to some embodiments of the present invention is shown. [Figure 3D] A schematic diagram of image content according to some embodiments of the present invention is shown. [Figure 3E] A schematic diagram of image content according to some embodiments of the present invention is shown. [Figure 3F] A schematic diagram of image content according to some embodiments of the present invention is shown.

[0013] [Figure 4] A schematic diagram of an exemplary architecture for interaction according to some embodiments of the present invention is shown.

[0014] [Figure 5] Shows a schematic diagram of an exemplary architecture for interaction according to some embodiments of the present invention.

[0015] [Figure 6] Shows a schematic structural block diagram of an exemplary device for interaction according to some embodiments of the present invention.

[0016] [Figure 7] Shows a block diagram of an electronic device of multiple embodiments in which the present invention can be implemented.

Embodiments for Implementing the Invention

[0017] Hereinafter, embodiments of the present invention will be described in more detail with reference to the drawings. Although specific embodiments of the present invention are shown in the drawings, the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Rather, these embodiments are provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not used to limit the protection scope of the present invention.

[0018] In the description of the embodiments of the present invention, the term "including" and its similar terms are open-ended inclusion of "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". There may be other explicit and implicit definitions in the following description.

[0019] In this specification, unless explicitly stated otherwise, performing one step "in response to A" does not mean immediately executing the step immediately after "A", but may include one or more intermediate steps.

[0020] It is understood that the data related to the technical solution (including but not limited to the data itself, data acquisition, or use) should comply with the corresponding laws and regulations and related designated requirements.

[0021] Before using the technical solutions disclosed in each embodiment of the present invention, the types, scope of use, usage scenarios, etc. of the information related to the present invention should be notified to the relevant users in an appropriate manner in accordance with the relevant laws and regulations, and the permission of the relevant users should be obtained.

[0022] For example, when responding to receiving an uncommitted request from a user, by sending prompt information to the relevant user, it is explicitly prompted to the user that the requested operation requires the acquisition and use of the user's personal information. Thereby, based on the prompt information, the user can independently select whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that executes the operation of the technical solution of the present invention.

[0023] As a selective but non-limiting implementation form, the method of sending prompt information to the user in response to receiving an uncommitted request from the user may, for example, be a method of using a pop-up window, and the prompt information can be displayed in the form of text within the pop-up window. In addition, the pop-up window may further include a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understood that the above-mentioned notification and user authorization acquisition process is only schematic and does not limit the embodiments of the present invention, and other methods that meet the relevant laws and regulations can also be applied to the embodiments of the present invention.

[0025] As used herein, the term "model" refers to a system that learns the relationships between corresponding inputs and outputs from training data and, after training is complete, can generate a corresponding output for a given input. Model generation can be based on machine learning techniques. Deep learning is a type of machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. Neural network models are an example of a model based on deep learning. In this specification, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably herein.

[0026] A neural network is a type of machine learning network based on deep learning. A neural network can process input and provide a corresponding output, and typically includes an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications usually include many hidden layers, thereby increasing the network's depth. Each layer of a neural network is connected sequentially so that the output of the previous layer is provided as the input of the subsequent layer, with the input layer receiving input to the neural network and the output of the output layer serving as the final output of the neural network. Each layer of a neural network contains one or more nodes (also called processing nodes or neurons), each node processing input from the previous layer.

[0027] Typically, machine learning is divided into three phases: the training phase, the testing phase, and the application phase (also called the inference phase). In the training phase, a specific model can be trained using a large amount of training data, and parameter values ​​are continuously and iteratively updated until the model can produce consistent inferences that meet the expected goals from the training data. Through training, the model is thought to be able to learn the relationships between inputs and outputs (also called input-to-output mappings) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether the model can provide the correct output, thereby determining the model's performance. The testing phase may also be integrated into the training phase. In the application or inference phase, the trained model can be used to process real model inputs based on the parameter values ​​obtained during training, and the corresponding model outputs can be determined.

[0028] As mentioned earlier, with the development of artificial intelligence technology, various types of product forms have emerged. For example, some AI products (e.g., digital assistants) provide users with human-computer interaction capabilities, allowing users to ask questions to the digital assistant via voice, text, etc., and the digital assistant can call upon machine learning models to generate answers, providing users with a great deal of convenience. However, these conventional AI products still have relatively low inference capabilities and accuracy for images, so there is still room for improvement in their performance in image-based human-computer interaction.

[0029] In view of this, embodiments of the present invention propose an improved solution for interaction. In this solution, when a user interaction request is received, interaction content including at least image content is collected using a content collection device. Object detection is performed on the image content to obtain an object detection result that can indicate at least one object within the image content. Based on the object detection result, points of interest are identified from at least one object. Determine the target object and interaction requirements for the target object, which are classified as Points of Interest (POIs). Then, perform the target operation associated with the target object based on the interaction requirements.

[0030] In this way, embodiments of the present invention can accurately identify a user's points of interest (POIs) in relation to image content, accurately determine the user's interaction requirements for the POIs, respond to the user's interaction requests based on those requirements, and improve the accuracy and quality of human-computer interaction based on images.

[0031] The following provides a more detailed explanation of various exemplary implementations of this solution, accompanied by diagrams.

[0032] (Example environment) Figure 1 shows a schematic diagram of an exemplary environment 100 that can realize an embodiment of the present invention. In this exemplary environment 100, an application is installed on a terminal device 110. A user 140 can interact with the application via the terminal device 110 and / or a device connected to the terminal device 110.

[0033] In embodiments of the present invention, the application can provide a digital assistant 120 that assists user 140 in task processing. The digital assistant 120 may have intelligent conversational and task processing capabilities. In some examples, the digital assistant 120 can receive interaction content from user 140, perform tasks based on inference capabilities, and provide responses. For example, the digital assistant 120 may support text conversation services, voice conversation services, image conversation services, and content conversations in other modes with user 140.

[0034] In some embodiments, the digital assistant 120 can collect user 140 interaction content using a content collection device 170. In some examples, the content collection device 170 may include an image collection unit 171 (e.g., a camera, webcam, scanner, etc.) and an audio collection unit 172 (e.g., a microphone). The digital assistant 120 can collect image content using the image collection unit 171 and audio content of the user 140 using the audio collection unit 172. The content collection device 170 may be deployed on or independent of the terminal device 110. Of course, the content collection device 170 is not limited to including an image collection unit 171 and an audio collection unit 172, and may further include other devices, and embodiments of the present invention are not limited thereto.

[0035] In some embodiments, the digital assistant 120 can use a machine learning model 160 (which may include one or more machine learning models, such as machine learning model 160-1, machine learning model 160-2, ..., machine learning model 160-N, where N is a positive integer; for convenience of explanation, one or more machine learning models are collectively referred to as machine learning model 160) to support interaction with the user 140. For example, the digital assistant can use one or more machine learning models 160 to provide a question-answering service to the user 140.

[0036] In environment 100, if the application is active, the terminal device 110 can present the application's user interface 150. The user interface 150 may include various pages that the application can provide, such as interaction pages between the user and the digital assistant 120. In some embodiments, the terminal device 110 can present interaction content 152 (including audio content, text content, image content, etc.) between the user 140 and the digital assistant 120 in the user interface 150.

[0037] The machine learning model 160 may be of a different type. In some embodiments, one or more machine learning models 160 may be built on a language model (LM). The machine learning model used is a content generation model that can generate corresponding outputs based on model inputs. In some embodiments, a machine learning model of a language model can receive model inputs in text format (e.g., natural language and / or machine language) and / or non-text format (e.g., images, audio, video, etc.) and generate desired outputs based on the model inputs and prompt words. The prompt words here are used to guide the machine learning model to generate user requirements that can be addressed by the model inputs, as indicated by the model inputs. In an application scenario to support user interaction, user 140's input may be provided to the machine learning model 160 as at least a portion of the model input (the other portion may include prompt words). The user input is considered a question. Based on the model output, a corresponding answer can be generated and provided to user 140.

[0038] In some embodiments, the terminal device 110 communicates with the service-side device 130 to provide services to the application. As shown in Figure 1, the service-side device 130 can call a machine learning model 160 to support human-computer interaction between the digital assistant 120 and the user 140 based on the output of the machine learning model 160. The terminal device 110 may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, locator devices, television receivers, radio receivers, e-book devices, game devices, or any combination thereof, and may include accessories, peripherals, or any combination thereof for these devices. In some embodiments, the terminal device 110 may also support any type of interface to the user (such as a “wearable” circuit).

[0039] The service-side device 130 may be any type of computing system / server capable of providing computing functions, including but not limited to mainframes, edge computing nodes, and computing devices in a cloud environment. The service-side device 130 may be implemented, for example, based on a cloud environment.

[0040] The structure and function of each element of Environment 100 are described for illustrative purposes only and should not be considered as limiting the scope of the present invention.

[0041] The following description will continue with reference to the attached drawings, illustrating some exemplary embodiments of the present invention.

[0042] (An example process) Figure 2 shows a flowchart of process 200 for interaction according to some embodiments of the present invention. For the sake of discussion, some embodiments of the present invention will be described below from the perspective of a terminal device 110, together with the environment 100 in Figure 1, but this is for illustrative purposes only. In some embodiments, the operations described with respect to the terminal device can be completed together by the terminal device in cooperation with the service-side device.

[0043] In box 210 of process 200, when terminal device 110 receives an interaction request from user 140 to digital assistant 120, it collects interaction content using content collection device 170. The interaction request is used to ask digital assistant 120 to perform a human-computer interaction. In some examples, terminal device 110 may present an icon of digital assistant 120. The user can trigger (e.g., click, long press, slide, etc.) the icon of digital assistant 120, and in response to detecting a trigger on the icon of digital assistant 120, terminal device 110 determines that an interaction request to digital assistant 120 has been received. In other examples, the user can also wake up digital assistant 120 via a voice command. Terminal device 110 may be configured to continuously detect sounds in the environment in which it is located. If it determines that the audio collected from the environment contains sound, it detects whether the sound contains a wake-up word to wake up digital assistant 120. If the terminal device 110 determines that the voice contains a wake-up word, it determines that an interaction request has been received for the digital assistant 120.

[0044] The interaction content includes at least image content. In some embodiments, the content collection device 170 may include an image collection unit 171 (e.g., a webcam, camera, or scanner). Upon receiving an interaction request to the digital assistant, the terminal device 110 controls the image collection unit 171 to collect image content. In some embodiments, the interaction content may further include audio content. The content collection device 170 may further include an audio collection unit 172 (e.g., a microphone). In response to receiving an interaction request to the digital assistant, the terminal device 110 controls the image collection unit 171 and the audio collection unit 172 to collect image content and audio content, respectively.

[0045] The content collection device 170 may be a component of the terminal device 110 or it may be independent of the terminal device 110. In one example, the content collection device 170 may include a camera and microphone deployed on the terminal device 110. In another example, the content collection device 170 may be deployed on an electronic device (e.g., glasses, headphones, etc.) that is connected to the terminal device 110. For example, the electronic device may have a camera and microphone, and the terminal device may be connected to the electronic device via a communication connection (e.g., Bluetooth® connection). Upon receiving a user interaction request to the digital assistant 120, the terminal device 110 may send a command to the electronic device via the communication connection to instruct the electronic device to turn on the camera and microphone and collect image and audio content, and receive image and audio content from the electronic device.

[0046] It should be noted that interaction content is not limited to image content and audio content, but may further include other forms of interaction content such as text content. Accordingly, the content collection device 170 is not limited to comprising an image collection unit 171 and an audio collection unit 172, but may further include collection units for collecting other forms of interaction content. The embodiments of the present invention do not limit the types of interaction content and content collection device 170.

[0047] In box 220 of process 200, the terminal device 110 performs object detection on the image content and obtains an object detection result. The object detection result indicates at least one object within the image content. Generally, various entities and regions within the image content can be identified as objects, so the at least one object may include various identifiable entities or regions within the image content. As an example, as shown in Figure 3A, Figure 3A shows a schematic diagram of example 300A of image content according to some embodiments of the present invention. The table lamp 302, sofa 302, hand 303, book 304, curtain 305, drawing 306, etc., in the image content shown in example 300A can be identified as objects.

[0048] In some embodiments, the object detection result may include at least one of the following: the location of at least one object within the image content, or at least one of the categories of at least one object. In some examples, the object detection result may include the object number, object mask (Object The object mask may include a bounding box (Bounding) that shows the object's contour and region. The object mask may point to the contour and region of the object. The category label indicates the category of the object. The category label may be selected from a pre-built set of category labels or may be determined based on the interaction content. Alternatively or additionally, the object detection result may include a bounding box (Bounding) that shows the contour and region of the object. It may also include a Box.

[0049] As an example, as shown in Figures 3A and 3B, Example 300B illustrates a method for segmenting the image content of Example 300A into multiple objects. In Example 300B, the table lamp 301 is identified as one object. The object detection result may include the bounding box 311, object number 312 (i.e., "3"), object mask (not shown), and category label (not shown) of the table lamp 301. The hand 303 is also identified as one object, and the object detection result further includes the bounding box 321 (i.e., "13"), object number 322, object mask, and category label of the hand 303. In practice, the object detection result may include the object number, bounding box, object mask, and category label, etc., for each object identified from the image content shown in Example 300A, and it should be understood that these are not listed one by one here. Also, note that Example 300B is merely an example provided to illustrate the solutions of embodiments of the present invention. In actual applications, objects in image content can be segmented and shown in any suitable way, and embodiments of the present invention are not limited thereto.

[0050] In some embodiments, the terminal device 110 can use a trained machine learning model to perform object detection on image content and obtain object detection results for the image content. As an example, as shown in Figure 4, Figure 4 shows a schematic diagram of an exemplary architecture 400 of a user interaction according to some embodiments of the present invention. The exemplary architecture 400 shows a machine learning model 160-1 (which may also be referred to herein as the first machine learning model). The terminal device 110 can generate a model input for the machine learning model 160-1 based on the image content 402. The terminal device 110 can provide the model input to the machine learning model 160-1 and obtain a model output generated by the machine learning model 160-1 based on the model input. The terminal device 110 can obtain object detection results 406 based on the model output. The machine learning model 160-1 can efficiently and accurately identify objects within the image content.

[0051] In some examples, as shown in Figure 5, Figure 5 shows a schematic diagram of an exemplary user interaction architecture 500 according to some embodiments of the present invention. In box 510, terminal device 110 detects a predetermined orientation formed by a directional guide object from image content 402. The guide object (sometimes called an interactor) can form a predetermined directional orientation and can support an object in space through the predetermined directional orientation. The guide object may include, but is not limited to, a hand, an eye, a pointer, a cursor on a display screen, or a point of light formed by a laser pen. The predetermined orientation may include various directional orientations that the guide object can form. For example, the guide object may include a user's hand, and the predetermined orientation may include a directional gesture.

[0052] In box 520, when it is detected that a given posture is a static posture, the terminal device 110 generates a model input for machine learning model 160-1 based on the static image in the image content. In box 520, when it is detected that a given state is a dynamic posture, the terminal device 110 generates a model input for machine learning model 160-1 based on the dynamic image in the image content. Subsequently, the terminal device 110 provides the model input to machine learning model 160-1 and obtains the object detection result 406 based on the model output of machine learning model 160-1. In this way, machine learning model 160-1 can identify the complete given posture and further divide each object in the image content based on the complete given posture, which helps to improve the accuracy of object detection.

[0053] In some embodiments, as shown in Figure 4, the terminal device 110 can also generate model input for the machine learning model 160-1 based on image content 402 and auxiliary prompt information 404 for the image content 402. The auxiliary prompt information 404 is used to help the machine learning model 160-1 understand objects in the image content, and can help the machine learning model 160-1 detect objects in the image content, thereby improving the accuracy of object detection.

[0054] In some examples, the auxiliary prompt information 404 may include historical interaction data. The historical interaction data can improve semantic completeness and can assist the machine learning model 160-1 in detecting objects within the image content 402, thereby improving the accuracy of object detection.

[0055] In some other examples, the interaction content may further include text content and / or audio content. The auxiliary prompt information 404 may include text content and / or audio content within the interaction content. If the auxiliary prompt information 404 includes audio content within the interaction content, the terminal device 110 can perform text recognition on the audio content to obtain a text prompt corresponding to the audio content. The text and audio content within the user's interaction content are typically associated with image content, which can help the machine learning model 160-1 understand objects within the image content 402, thereby improving the accuracy of object detection.

[0056] In some other examples, the auxiliary prompt information may further include image prompts. For example, an image prompt may include an image of one or more specific objects. Such an image prompt can guide the machine learning model 160-1 to detect objects that are the same as or similar to the one or more specific objects from the image content. Alternatively, for example, an image prompt may include one or more prompt images and the object detection results of the one or more prompt images. In this way, the machine learning model 160-1 can be guided to perform object segmentation on the image content 402 according to a similar object segmentation method.

[0057] In box 230 of process 200, the terminal device 110 determines, based on the object detection results, a target object and interaction requirements for the target object, which are classified as POIs from at least one object. A POI may be understood as an object of interest to the user within the image content, or as an object related to interaction. Interaction requirements are used to indicate the user's expectations or requirements for interaction with the digital assistant 120. For example, interaction requirements may indicate responses related to a POI that the user expects to provide feedback to the digital assistant 120, or actions related to a POI that the user expects the digital assistant 120 to perform.

[0058] In some examples, as shown in Figure 4, the terminal device 110 can generate a model input (i.e., a second model input) for a machine learning model 160-2 (which may also be referred to herein as a second machine learning model) based on the target detection result 406. The terminal device 110 can provide the model input to the trained machine learning model 160-2 and obtain the model output (i.e., a second model output) generated by the machine learning model 160-2. Subsequently, the terminal device 110 can determine the target object and interaction requirements 412 based on the model output of the machine learning model 160-2. The machine learning model 160-2 can efficiently and accurately determine the POI and interaction requirements.

[0059] In some embodiments, the terminal device 110 can determine the relative position and relationship between at least one object based on the object segmentation results. The terminal device 110 can then determine target objects and interaction requirements based on the relative position and relationship between at least one object. For example, the terminal device 110 can generate model input for machine learning model 160-2 based on a prompt word, object mask, and category label. The prompt word can instruct machine learning model 160-2 to analyze the relative position and relationship between multiple objects within the image content based on the object mask and category label. For example, the prompt word can instruct machine learning model 160-2 to analyze the relative distance, spatial arrangement, relative direction, interaction relationship, etc., between multiple objects. The prompt word can further instruct machine learning model 160-2 to output the user's POIs and interaction requirements based on the analysis results. Based on the model output of machine learning model 160-2, the terminal device 110 can determine target objects classified as POIs and interaction requirements for those POIs.

[0060] In some embodiments, the interaction content further includes user voice content and / or text content, and the terminal device 110 can form a target object by determining, based on the object detection result, an object associated with the voice content and / or text content from at least one object. As an example, as shown in Figure 4, the terminal device 110 can acquire text content 408 and / or voice content 410 within the interaction content. Based on the object detection result 406, text content 408 and / or voice content 410, and a prompt word, the terminal device 110 generates model input for the machine learning model 160-2. The prompt word can instruct the machine learning model 160-2 to analyze the relative positions and relationships between multiple objects in the image content based on the object mask, category label in the object detection result 406, text content 408 and / or voice content 410, and to determine POIs and interaction requirements based on the analysis results. Subsequently, the terminal device 110 can determine the target object and interaction requirements classified as POIs based on the model output of the machine learning model 160-2. As another example, the terminal device 110 can also generate model input for the machine learning model 160-2 based on the target detection result 406, text content 408 and / or audio content 410, historical interaction content, and prompt words. In this way, it is possible to improve semantic completeness and further enhance the accuracy of POI and interaction requirements.

[0061] In some embodiments, the at least one object may include multiple objects. The terminal device 110 can determine a directional guide object from the multiple objects based on the object detection result. The object indicated by the guide object among the multiple objects is determined as the target object. As noted in the above analysis, the guide object may include, but is not limited to, a hand, eye, pointer, cursor on a display screen, or a light point formed by a laser pen. In some examples, the terminal device 110 can determine the guide object and the direction in which the guide object is indicated based on the object mask and category label. The object corresponding to the direction in which the guide object is indicated among the multiple objects is determined as the target object. In this way, the interaction methods supported by the digital assistant 120 are enriched, and the user is not limited to interacting with the digital assistant 120 via text or voice, but can also interact with the digital assistant 120 synchronously via, for example, body movements or pointing tools, thereby improving the flexibility and versatility of interaction with the digital assistant 120.

[0062] In some embodiments, the terminal device 110 determines at least one predetermined posture formed by the guide target based on the target detection result. The target associated with at least one predetermined posture among a plurality of targets is determined as the target target. The predetermined posture may include postures with various orientations that the guide target can form. In practical applications, some predetermined postures that the guide target can form can be determined in advance, and a predetermined set of postures can be constructed based on these predetermined postures. The terminal device 110 can determine, based on the target detection result, whether the guide target has formed a posture from the predetermined set of postures. In this way, it helps maintain consistency in the interaction method.

[0063] As an example, Figure 3C shows a schematic diagram of Example 300C of image content according to some embodiments of the present invention. With respect to the image content shown in Example 300C, the terminal device 110 can determine the hand 303 in the image content as the guide target. Assume that the user's hand 303 is forming a circle selection gesture 331 (i.e., a predetermined posture). The terminal device 110 can determine the target object based on the direction of the hand 303's indication and the circle selection range of the circle selection gesture 331, and for example, it can determine the table lamp 301 located in the direction of the hand 303's indication and corresponding to the circle selection range of the circle selection gesture 331 as the target object.

[0064] As another example, Figure 3D shows a schematic diagram of example 300D of image content relating to some embodiments of the present invention. In example 300D, the user's hand 303 forms a smearing gesture 341 that belongs to a pose within a predetermined set of poses. Based on the direction of the hand 303 and the smearing gesture range of the smearing gesture 341, the terminal device 110 can determine, for example, a table lamp 301 as the target object.

[0065] As another example, Figure 3E shows a schematic diagram of Example 300E of image content according to some embodiments of the present invention. With respect to the image content shown in Example 300D, the terminal device 110 can determine the user's hands 351, 352 as the guide target. The user's hands 351, 352 form a frame selection gesture that may belong to a pose within a predetermined set of poses. Based on the direction of indication and the frame selection range of the frame selection gesture, the terminal device 110 can determine, for example, a table lamp 301 as the target target.

[0066] In the examples described above, the solutions of the embodiments of the present invention were interpreted and explained using the user's hand as the guide target, but it should be noted that the guide target is not limited to the hand. Depending on the actual needs, an object with an appropriate orientation can be selected as the guide target. Furthermore, the predetermined posture is not limited to the posture described above, and any appropriate predetermined posture can be constructed based on the selected guide target. The embodiments of the present invention do not limit the type of guide target or the type of predetermined posture.

[0067] In some embodiments, the terminal device 110 can determine a gaze region within the image content that the user is focusing on, and determine a target object based on one or more objects located within the gaze region. As an example, Figure 3F shows a schematic diagram of example 300F of image content according to some embodiments of the present invention. The image content shown in example 300F includes a user image 361. The terminal device 110 can determine the user's posture information (e.g., information indicating the user's head posture) based on the user image 361. Based on the user's posture information, the terminal device 110 can determine a gaze region 362 within the image content that the user is focusing on, and determine a table lamp 301 located in the gaze region 362 as a target object. In this way, interaction between the user and the digital assistant 120 based on image content can be combined with visual detection, further improving the flexibility and diversity of the interaction.

[0068] In some embodiments, the terminal device 110 can determine the gaze area in the image content that the user is looking at, based on at least one of the following: configuration information of the content collection device 170, orientation information of the content collection device 170, or user eye movement information. The configuration information may include various device information related to the determination of the gaze area; for example, the content collection device 170 may include the device type, internal parameters and external parameters of the image collection unit 171, etc. The orientation information can indicate the position and orientation of the content collection device 170 in space; for example, the orientation information may include the pitch angle, roll angle, or yaw angle of the camera, etc. The configuration information, orientation information, and eye movement information allow for the accurate determination of the gaze area in the image content that the user is looking at, and furthermore, allow for the accurate determination of POIs and interaction requirements.

[0069] As an example, the content collection device 170 may include glasses. The glasses may be equipped with a camera. Configuration information may include information about the type of glasses, the camera's internal parameters and external parameters, etc. The terminal device 110 can generate model input for the machine learning model 160-2 based on prompt words, configuration information, target mask, and category labels. Prompt words can instruct the machine learning model 160-2 to analyze the relative positional relationships between multiple objects and the relative positional relationship between the user's eyes and the camera. The prompt words can further instruct the machine learning model 160-2 to determine the gaze region in the image that the user's eyes are fixated on, based on the relative positional relationships between multiple objects and the relative positional relationship between the eyes and the camera, and further to determine a target object based on one or more objects located within the gaze region.

[0070] As another example, the content collection device 170 may include glasses. The glasses may be equipped with a camera and an eye-motion tracker. The camera can be used to collect image content of the environment, and the eye-motion tracker can be used to collect eye-movement tracking data of the user's eyes. The terminal device 110 can generate model input for machine learning model 160-2 based on the eye-movement tracking data and object detection results, and can use machine learning model 160-2 to determine one or more gaze points (which may also be gaze regions) that the user is looking at within the image content. The terminal device 110 can determine a target object based on one or more objects corresponding to the one or more gaze points.

[0071] As another example, the content collection device 170 may include wireless headphones. The wireless headphones may be equipped with a camera and an attitude sensor. Image content can be collected using the camera, and attitude information of the wireless headphones can be collected using the attitude sensor. The terminal device 110 can acquire configuration information of the wireless headphones (e.g., information indicating the placement position of the wireless headphones, internal and external parameters of the camera, etc.), attitude information, and the collected image content. Based on the configuration information, attitude information, and target detection results, the terminal device 110 can generate model inputs for the machine learning model 160-2, and can also instruct the machine learning model 160-2 to analyze the relative positional relationship between the user's eyes and the camera and the relative positional relationships between multiple objects based on the configuration information and attitude information. Subsequently, the terminal device 110 outputs a model output that includes POIs and interaction requirements based on the relative positional relationship between the eyes and the camera and the relative positional relationships between multiple objects. Based on the model output, the terminal device 110 can determine target objects classified as POIs and interaction requirements for the POIs.

[0072] In box 240 of process 200, the terminal device 110 uses the digital assistant 120 to perform a target operation associated with the target object based on the interaction requirements. In some embodiments, the terminal device 110 can determine at least one predetermined command that indicates the interaction requirements, and then use the digital assistant 120 to perform a target operation based on that at least one predetermined command. For example, a predetermined set of commands can be pre-built. The terminal device 110 can generate model inputs for the machine learning model 160-2 based on prompt words, object masks, and category labels. The prompt words can instruct the machine learning model 160-2 to analyze the relative positions and relationships between multiple objects, predict the user's points of interest (POIs) for image content based on the analysis results, and select one or more matching predetermined commands from the predetermined set of commands based on the analysis results. The terminal device 110 can perform a target operation based on one or more predetermined commands in the model output.

[0073] In some embodiments, the terminal device 110 determines at least one predetermined posture formed by a directional guide target among at least one target, based on the target detection result. Then, it determines at least one predetermined command associated with the at least one predetermined posture from a plurality of candidate predetermined commands. For example, a mapping relationship between a predetermined command and a predetermined posture can be predetermined, for instance, predetermined commands corresponding to the circle selection gesture 331 shown in Figure 3C, the smearing gesture gesture 341 shown in Figure 3D, and the frame selection gesture shown in Figure 3E can be predetermined. Once at least one predetermined posture formed by a guide target is determined, the terminal device 110 can determine at least one predetermined command to which that at least one predetermined posture is mapped, based on the mapping relationship between the predetermined command and the predetermined posture.

[0074] It is understood that the target operation here may include various operations that can be performed by the digital assistant 120. In some embodiments, the target operation may include operations that answer user questions. Specifically, the terminal device 110 can generate answers to user questions related to the target object. For example, the user may point to the table lamp 301 with the gesture shown in Figure 3A and ask questions by voice or text, such as "What is this?" or "What brand is this?". Based on the prompt word, object mask, category label, and user question, the terminal device 110 can generate model input for the machine learning model 160-2. The machine learning model 160-2 can also generate an answer to the user question while determining that the POI is the table lamp 301. The terminal device 110 can present the answer or play an audio answer, such as "This is a table lamp" or "This is a table lamp of the XXX brand."

[0075] In some embodiments, the digital assistant 120 can control a target device classified as a POI to perform a target operation. Specifically, the terminal device 110 can determine, based on the target detection result, a target device associated with the digital assistant 120 and a control command indicating the interaction requirements for the target device from at least one target. Here, the target device associated with the digital assistant 120 may include devices on which the digital assistant 120 has control authority, such as household appliances such as televisions, air conditioners, refrigerators, water heaters, and table lamps bound to the digital assistant 120, or wearable devices such as headphones and glasses connected to the terminal device 120. The terminal device 110 can use the digital assistant 120 to send a control command to the target device and instruct the target device to perform a target operation based on the control command.

[0076] For example, the user can point to the table lamp 301 with the gesture shown in Figure 3A and issue commands such as "turn it off" or "increase the brightness" by voice or text. The terminal device 110 can provide the user's voice or text content and the object detection results of the image content to the machine learning model 160-2, determine the POI as the table lamp 301 using the machine learning model 160-2, and obtain the "turn off command" or "brightness adjustment command" generated by the machine learning model 160-2. Based on the communication connection between the terminal device 110 and the table lamp 301, the digital assistant 120 can turn off the table lamp 301 or adjust the brightness of the table lamp 301 by sending the "turn off command" or "brightness adjustment command" to the table lamp 301.

[0077] In this way, embodiments of the present invention can accurately identify a user's points of interest (POIs) in relation to image content, accurately determine the user's interaction requirements for the POIs, respond to the user's interaction requests based on those requirements, and improve the accuracy and quality of human-computer interaction based on images.

[0078] (Examples of devices and apparatus) Embodiments of the present invention further provide corresponding apparatuses for implementing the above method or process. Figure 6 shows a schematic structural block diagram of an exemplary apparatus 600 for interaction according to a particular embodiment of the present invention. The apparatus 600 may be implemented as a terminal device 110 or included within the terminal device 110. Each module / component within the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0079] As shown in Figure 6, the device 600 comprises a collection module 610, an acquisition module 620, a decision module 630, and an execution module 640. The collection module 610 is configured to collect interaction content using a content collection device in response to receiving a user interaction request, and the interaction content includes at least image content. The acquisition module 620 is configured to perform object detection on the image content to obtain object detection results, which indicate at least one object within the image content. The decision module 630 is configured to determine, based on the object detection results, a target object classified as a point of interest and interaction requirements for the target object from at least one object. The execution module 640 is configured to perform target operations associated with the target object based on the interaction requirements.

[0080] In some embodiments, the object detection result includes at least one of the following: the location of at least one object within the image content, or at least one category of the object.

[0081] In some embodiments, at least one object includes multiple objects, and the determination module 630 is further configured to determine a directional guide object from the multiple objects based on the object detection results, and to determine the object indicated by the guide object among the multiple objects as the target object.

[0082] In some embodiments, the determination module 630 is further configured to determine at least one predetermined posture formed by the guide object based on the object detection result, and to determine the object associated with at least one predetermined posture among the multiple objects as the target object.

[0083] In some embodiments, the determination module 630 is further configured to determine a gaze region within the image content that the user is looking at, and to determine a target object based on one or more objects located within the gaze region.

[0084] In some embodiments, the interaction content further includes user voice content and / or text content, and the decision module 630 is further configured to determine, based on the target detection results, a target object associated with the voice content and / or text content from at least one object, and interaction requirements for the target object.

[0085] In some embodiments, the acquisition module 620 is further configured to generate a first model input for a first machine learning model based on at least the image content, and to obtain a target detection result based on the first model output determined by the first machine learning model for the first model input.

[0086] In some embodiments, the acquisition module 620 is further configured to generate a first model input based on image content and auxiliary prompt information for the image content.

[0087] In some embodiments, the acquisition module 620 is further configured to detect a predetermined pose formed by a directional guide object from the image content, generate a first model input based on a static image in the image content in response to the detected predetermined pose being a static gesture, and generate a first model input of a dynamic image in the image content in response to the detected predetermined pose being a dynamic gesture.

[0088] In some embodiments, the decision module 630 is further configured to generate a second model input for a second machine learning model based on the target detection results, and to determine the target object and interaction requirements based on the second model output input by the second machine learning model to the second model.

[0089] In some embodiments, the decision module 630 is further configured to determine at least one predetermined instruction that indicates an interaction requirement, and the execution module 640 is further configured to perform a target operation based on at least one predetermined instruction.

[0090] In some embodiments, the decision module 630 is further configured to determine at least one predetermined posture formed by a directional guide object among at least one object based on the object detection result, and to determine at least one predetermined command associated with at least one predetermined posture from a plurality of predetermined command candidates.

[0091] In some embodiments, the decision module 630 is further configured to determine, based on the object detection results, a target device and a control command indicating interaction requirements for the target device from at least one object, and the execution module 640 is further configured to send the control command to the target device, instructing the target device to perform the target operation based on the control command.

[0092] The units and / or modules included in the device 600 may be implemented using a variety of methods, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to, or instead of, machine-executable instructions, some or all units and / or modules in the device 600 may be implemented at least partially by one or more hardware logic components. Exemplary types of hardware logic components that may be used, but not limited to, include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on a chip (SOCs), and composite programmable logic devices (CPLDs).

[0093] Figure 7 shows a block diagram of an electronic device 700 that can carry out one or more embodiments of the present invention. It should be understood that the electronic device 700 shown in Figure 7 is illustrative and should not limit the function and scope of the embodiments described herein. The electronic device 700 shown in Figure 7 may include the terminal device 110 of Figure 1 or the apparatus 600 of Figure 6, or may be implemented as the terminal device 110 of Figure 1 or the apparatus 600 of Figure 6.

[0094] As shown in Figure 7, the electronic device 700 is in the form of a general-purpose electronic device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage devices 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and can perform various processes based on a program stored in memory 720. In a multiprocessor system, the parallel processing capability of the electronic device 700 is improved by having multiple processing units execute computer executable instructions in parallel.

[0095] The electronic device 700 typically includes multiple computer storage media. Such media may include, but are not limited to, volatile and non-volatile media, removable and non-removable media, and may be any obtainable media accessible by the electronic device 700. Memory 720 may be volatile memory (e.g., registers, fast cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or a specific combination thereof. Storage device 730 may be removable or non-removable media, and may include machine-readable media such as flash memory drives, magnetic disks, or any other media, which may be used to store information and / or data and may be accessible within the electronic device 700.

[0096] The electronic device 700 may further include other removable / non-removable, volatile / non-volatile storage media. Although not shown in Figure 7, a magnetic disk drive for reading from or writing to removable, non-volatile magnetic disks (e.g., “floppy disks”) and a removable optical disk drive for reading from or writing to non-volatile optical disks may be provided. In these cases, each drive may be connected to a path (not shown) by one or more data medium interfaces. The memory 720 may also include a computer program product 725 having one or more program modules, which are configured to perform various methods or operations of various embodiments of the present invention.

[0097] The communication unit 740 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 may be implemented as a single computing cluster or multiple computing machines, which can communicate via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.

[0098] The input device 750 may be one or more input devices such as a mouse, keyboard, or trackball. The output device 760 may be one or more output devices such as a display, speaker, or printer. The electronic device 700 may further communicate with one or more external devices (not shown), such as a storage device or display device, via the communication unit 740 as needed, or with one or more devices that enable a user to interact with the electronic device 700, or with any device (e.g., a netbook card, modem, etc.) that enables the electronic device 700 to communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0099] An exemplary embodiment of the present invention provides a computer-readable storage medium in which computer-executable instructions are stored, and the computer-executable instructions are executed by a processor to realize the above method. An exemplary embodiment of the present invention further provides a computer program product, the computer program product including computer-executable instructions, which is tangibly stored on a non-temporary computer-readable medium, and the computer-executable instructions are executed by a processor to realize the above method.

[0100] Herein, each aspect of the present invention has been described with reference to flowcharts and / or block diagrams of methods, apparatus, devices, and computer program products realized by the present invention. It should be understood that each box in the flowcharts and / or block diagrams, and each combination of boxes in the flowcharts and / or block diagrams, may be realized by computer-readable program instructions.

[0101] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device to generate a machine that, when these instructions are executed by the computer or other programmable data processing device's processing unit, generates a device for performing functions / operations specified in one or more boxes in a flowchart and / or block diagram. These computer-readable program instructions may be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other device to operate in a particular manner so that the computer-readable medium on which the instructions are stored constitutes a product containing instructions for performing each aspect of the functions / operations specified in one or more boxes in a flowchart and / or block diagram.

[0102] By loading computer-readable program instructions into a computer, other programmable data processing device, or other device, a series of operational steps are performed on the computer, other programmable data processing device, or other device to generate a computer-implemented process, thereby enabling the instructions executed on the computer, other programmable data processing device, or other device to perform a function / operation specified in one or more boxes in a flowchart and / or block diagram.

[0103] The flowcharts and block diagrams in the drawings illustrate the implementable architectures, functions, and operations of systems, methods, and computer program products of multiple implementations according to the present invention. In this regard, each box in a flowchart or block diagram may represent a module, program segment, or part of an instruction, and a module, program segment, or part of an instruction contains one or more executable instructions for implementing a specified logical function. In some implementations as replacements, the functions represented in the boxes may occur in a different order than those shown in the drawings. For example, two consecutive boxes may actually be executed substantially in parallel, or in reverse order depending on the functions involved. It should also be noted that each box in a block diagram and / or flowchart, and combinations of boxes in a block diagram and / or flowchart, may be implemented by a special-purpose hardware-based system that performs a specified function or operation, or by a combination of special-purpose hardware and computer instructions.

[0104] While the various realizations of the present invention have been described above, the above descriptions are illustrative, not exhaustive, and not limited to the realizations disclosed. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of the realizations described. The choice of terms used herein is intended to best interpret the principle, practical application, or improvement to the art in the market of each realization, or to enable those skilled in the art to understand each realization disclosed herein.

Claims

1. A method for interaction, In response to receiving a user interaction request, the interaction content is collected using a content collection device, wherein the interaction content includes at least image content. The process involves performing object detection on the aforementioned image content and obtaining an object detection result, wherein the object detection result indicates at least one object within the image content. Based on the object detection results, determine a target object classified as a point of interest and interaction requirements for that target object from among the at least one object. This includes performing a target operation associated with the target object based on the interaction requirements, The aforementioned method.

2. The object detection result includes the location of the at least one object within the image content, or at least one of the categories of the at least one object. The method according to claim 1.

3. The aforementioned at least one object includes a plurality of objects, Determining the target object from the aforementioned at least one object is, Based on the target detection results, a guide target with directionality is determined from the plurality of targets. This includes determining the target object among the plurality of objects that is indicated by the guide object as the target object, The method according to claim 1.

4. Determining the target object as the object indicated by the guide object among the at least one of the aforementioned objects is: Based on the object detection results, determine at least one predetermined posture formed by the guide object. This includes determining the target object as the object associated with at least one predetermined posture among the plurality of objects, The method according to claim 3.

5. Determining the target object from the aforementioned at least one object is, Determining the gaze area within the image content that the user is focusing on, This includes determining the target object based on one or more objects located within the gaze area, The method according to claim 1.

6. The interaction content further includes the user's voice content and / or text content. Determining the target object and the interaction requirements for the target object is Based on the target detection results, the process includes determining, from the at least one target, a target target associated with the audio content and / or the text content, and the interaction requirements for the target target. The method according to claim 1.

7. Performing object detection on the aforementioned image content and obtaining the object detection result is, To generate a first model input for a first machine learning model based at least on the image content, This includes obtaining the target detection result based on the first model output determined by the first machine learning model for the first model input, The method according to claim 1.

8. Generating the first model input for the first machine learning model is The process includes generating the first model input based on the image content and auxiliary prompt information for the image content, The method according to claim 7.

9. Generating the first model input for the first machine learning model is From the aforementioned image content, a predetermined posture formed by a directional guide object is detected, In response to the detected predetermined posture being a static gesture, the first model input is generated based on the static image in the image content, The process includes generating the first model input based on the dynamic image in the image content in response to the detected predetermined posture being a dynamic gesture, The method according to claim 7.

10. Determining the target object and the interaction requirements for the target object from at least one of the aforementioned objects is: Based on the target detection results, a second model input for the second machine learning model is generated, This includes determining the target object and the interaction requirements based on the output of the second model input by the second machine learning model to the second model, The method according to claim 1.

11. Determining the interaction requirements for the aforementioned target object is This includes determining at least one predetermined instruction that indicates the interaction requirement, Performing the aforementioned target operation means, The process includes performing the target operation based on at least one predetermined command. The method according to claim 1.

12. Determining at least one predetermined instruction that indicates the aforementioned interaction requirement is: Based on the object detection result, determine at least one predetermined posture formed by a directional guide object among the at least one object, This includes determining the at least one predetermined command associated with the at least one predetermined posture from a plurality of candidate predetermined commands, The method according to claim 11.

13. Determining the target object and the interaction requirements for the target object is Based on the target detection result, the process includes determining a target device and a control command indicating interaction requirements for the target device from at least one of the targets, Performing the aforementioned target operation means, This includes transmitting the control command to the target device and instructing the target device to perform the target operation based on the control command, The method according to claim 1.

14. A device for interaction, A collection module configured to collect interaction content using a content collection device in response to receiving a user interaction request, wherein the interaction content includes at least image content, and the collection module... An acquisition module configured to perform object detection on the image content and obtain an object detection result, wherein the object detection result indicates at least one object within the image content. A decision module configured to determine, based on the object detection results, a target object classified as a point of interest from the at least one object and interaction requirements for the target object, A system comprising: an execution module configured to perform a target operation associated with the target object based on the interaction requirements; The aforementioned device.

15. It is an electronic device, At least one processing unit, The device comprises at least one memory coupled to the at least one processing unit and storing instructions to be executed by the at least one processing unit, wherein, when the instructions are executed by the at least one processing unit, the electronic device causes the electronic device to perform the method according to any one of claims 1 to 13. The aforementioned electronic device.

16. A computer program is stored, and the computer program is executed by a processor to realize the method according to any one of claims 1 to 13. Computer-readable storage medium.

17. Computer program products, including computer programs, When the computer program is executed by the processor, the method according to any one of claims 1 to 13 is realized. The aforementioned computer program product.

Images