An article recommendation method and electronic device

Abstract

The application discloses an article recommendation method and an electronic device. The method comprises the following steps: obtaining position information and function information of a candidate article, wherein the function information at least represents a task that can be processed by the corresponding candidate article, and the candidate article comprises an article in a space environment where the electronic device is located; determining a recommended article from the candidate article according to obtained user intention information and the function information of the candidate article, wherein the user intention information at least represents a task that needs to be processed by the user currently; and performing a prompt operation for indicating a position of the recommended article based on the position information of the recommended article.

Description

Technical Field

[0001] This application relates to the field of recommendation technology, and in particular to a method for recommending items and an electronic device. Background Technology

[0002] In daily work and life, users often need to find the items they need from a variety of randomly placed items. For example, they need to find a pen and notebook when they are in a meeting, a water cup when they are making tea in the break room, and a USB flash drive when they are printing materials.

[0003] When there are too many items in the environment (such as a desktop environment) and they are not arranged in a good order, users will have difficulty finding the items they need in a timely manner. Summary of the Invention

[0004] Therefore, this application discloses the following technical solution:

[0005] The first aspect of this application provides a method for recommending items, applied to electronic devices, including:

[0006] Obtain the location and functional information of candidate items, wherein the functional information at least characterizes the task that the corresponding candidate item can handle, and the candidate items include items within the spatial environment where the electronic device is located;

[0007] Recommended items are determined from the candidate items based on the obtained user intent information and the functional information of the candidate items, wherein the user intent information at least represents the task that the user currently needs to process;

[0008] Based on the location information of the recommended item, a prompting operation is performed to indicate the location of the recommended item.

[0009] Optionally, obtaining the location and functional information of the candidate items includes:

[0010] Scan the spatial environment in which the electronic device is located to obtain at least one frame of environmental image; wherein, the spatial environment includes the desktop environment;

[0011] The location and type information of items within the desktop environment are obtained by identifying at least one frame of the environmental image.

[0012] The functional information of the candidate items is determined based on the type information of the candidate items.

[0013] Optionally, the step of performing a prompting operation based on the location information of the recommended item to indicate the location of the recommended item includes at least one of the following:

[0014] Based on the location information of the recommended item, the projection module of the electronic device is controlled to project onto the location of the item;

[0015] Output prompt information determined based on the location information of the recommended items. The prompt information may be voice information, text information, or image information.

[0016] Optional, also includes:

[0017] Obtaining the user's intent information specifically includes at least one of the following:

[0018] The user intent information is obtained based on the user's behavior.

[0019] The user intent information is obtained based on the user's input, which can be either voice input or text input.

[0020] When a user's behavior meets the triggering conditions, the user intent information is obtained based on the user's input.

[0021] Optionally, determining recommended items from the candidate items based on the obtained user intent information and the functional information of the candidate items includes:

[0022] Obtain the user's usage record information, which includes items used by the user;

[0023] Based on the usage record information, the user intent information, and the functional information of the candidate items, a recommended item is determined from the candidate items.

[0024] Optionally, it may also include at least one of the following:

[0025] Trigger the electronic device and / or other electronic devices to launch the target application, the target application being able to handle the same tasks as the recommended item;

[0026] Switch the electronic device from its current operating mode to a target operating mode. The electronic device in the target operating mode is capable of handling the same tasks as the recommended item.

[0027] Optionally, determining recommended items from the candidate items based on the obtained user intent information and the functional information of the candidate items includes:

[0028] If at least one candidate item's matching degree meets the recommendation criteria, the candidate item whose matching degree meets the recommendation criteria is determined as a recommended item. The matching degree of the candidate item is determined based on the user intent information and the functional information of the candidate item.

[0029] If the matching degree of each candidate item does not meet the recommendation conditions, obtain the location information and functional information of the updated candidate items.

[0030] Recommended items are determined from the updated candidate items based on the user intent information and the functional information of the updated candidate items.

[0031] Optionally, obtaining the location information and functional information of the candidate items includes at least one of the following:

[0032] In response to obtaining at least one of the user's user behavior and user input, the location information and functional information of the candidate item are obtained, wherein at least one of the user behavior and user input is used to determine the user intent information;

[0033] The location and functional information of candidate items are obtained based on the time period;

[0034] In response to any change in the quantity, type, orientation, or position of items in the spatial environment where the electronic device is located, the location and functional information of the candidate items are obtained.

[0035] In response to the user's trigger command, obtain the location and functional information of the candidate items.

[0036] A second aspect of this application provides an electronic device, including a data acquisition module, a memory, and a processor;

[0037] The acquisition module is used to detect items in the spatial environment where the electronic device is located;

[0038] The memory is used to store computer programs;

[0039] The processor is used to execute the computer program to perform:

[0040] Obtain the location and functional information of candidate items, wherein the functional information at least characterizes the task that the corresponding candidate item can handle, and the candidate items include items within the spatial environment where the electronic device is located;

[0041] Recommended items are determined from the candidate items based on the obtained user intent information and the functional information of the candidate items, wherein the user intent information at least represents the task that the user currently needs to process;

[0042] Based on the location information of the recommended item, a prompting operation is performed to indicate the location of the recommended item.

[0043] Optional, also includes:

[0044] The projection module is used to project onto any location;

[0045] The processor is specifically used to control the projection module to project onto the location of the recommended item based on the location information of the recommended item;

[0046] Or, it may also include:

[0047] The output module is used to output the prompt information determined by the processor based on the location information of the recommended item. The prompt information may be voice information, text information, or image information. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0049] Figure 1 This is a flowchart of an item recommendation method provided in an embodiment of this application;

[0050] Figure 2 This is an example diagram illustrating an application scenario of an item recommendation method provided in an embodiment of this application;

[0051] Figure 3 This is a flowchart illustrating a method for obtaining location and functional information of candidate items, as provided in an embodiment of this application.

[0052] Figure 4 This is a flowchart of another item recommendation method provided in an embodiment of this application;

[0053] Figure 5 This is a flowchart of another item recommendation method provided in the embodiments of this application;

[0054] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0055] Figure 7 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] This embodiment provides a method for recommending items, applied to electronic devices. Please refer to [link / reference]. Figure 1 The method may include the following steps.

[0058] S101, obtain the location information and functional information of the candidate items. The functional information at least represents the task that the corresponding candidate item can handle. The candidate items include items in the spatial environment where the electronic device is located.

[0059] S102, determine recommended items from candidate items based on the obtained user intent information and the functional information of candidate items, wherein the user intent information at least represents the task that the user needs to process at present.

[0060] S103, based on the location information of the recommended item, perform a prompting operation to indicate the location of the recommended item.

[0061] After each execution of S101, steps S102 and S103 can be executed only once, or multiple rounds of steps S102 and S103 can be executed based on multiple user intent information obtained. In other words, the location and function information of the candidate item can be obtained once, and then the recommended item can be determined and the prompt operation can be executed multiple times based on the location and function information obtained this time.

[0062] The beneficial effect of this embodiment is that, based on the task that the user needs to process and the tasks that each candidate item can process, the recommended item whose functional information matches the task that the user needs to process is determined from the candidate items, and then a prompting operation is performed to indicate the current location of the recommended item to the user. Thus, the electronic device of this embodiment can understand the user's intention in a timely manner and assist the user in finding the required item in a short time based on the user's intention, thereby improving the convenience of the user when finding items.

[0063] The execution entity of the item recommendation method in this embodiment can be any electronic device capable of collecting information about surrounding items and performing operations to indicate the location of items. As examples, this electronic device can be a desktop robot that can be placed on a desktop (such as an office desk). The appearance of the desktop robot is not limited; for example, it can have... Figure 2 As shown in the appearance of device 200 in (1), the end 202 of device 200 can rotate in space under the drive of the support body 201, and the end 202 can include any one or more of the acquisition module, projection module and output module integrated inside the end 202, so as to realize related functions through these modules.

[0064] For ease of distinction, the following description will use a desktop robot as an example to illustrate the method of this embodiment.

[0065] Desktop robots can obtain information about objects by scanning them within a certain range using a data acquisition module. The data acquisition module can generally only scan a limited range, such as within 1 meter. Correspondingly, the spatial environment in which electronic devices are located can be the spatial environment that the desktop robot can scan.

[0066] Desktop robots can perform prompting operations through their own projection and / or output modules. However, the prompting operations performed through the projection and / or output modules can only affect a limited range around the desktop robot. For example, when projecting through the projection module, if the distance is too far, the projected beam will weaken to the point of being invisible. Therefore, the spatial environment in which the electronic device is located can also be the spatial environment in which the desktop robot can perform prompting operations. The spatial environment in which prompting operations can be understood as the spatial environment that the performed prompting operations can affect.

[0067] Alternatively, the spatial environment in which the electronic device is located can be the intersection or union of the spatial environment in which prompting operations can be performed and the spatial environment that can be scanned.

[0068] Combination Figure 2 In the application scenario example of (1), when the desktop robot is placed on the desktop of any desk in the office, the spatial environment of the electronic device in S101 can be the desktop environment of this desk. In this case, the candidate items can include items within the desktop environment, that is, all items placed on the desktop of this desk. Figure 2 For example (1), candidate items may include: water cup A01, pen A02, stylus A03, remote control A04, notebook A05.

[0069] In this embodiment, the desktop robot can identify candidate items currently existing in the spatial environment where the electronic device is located through the acquisition module, and then obtain the location information and functional information of each candidate item.

[0070] Optional, see Figure 3 Obtaining the location and functional information of candidate items may include the following steps.

[0071] S301, Scan the spatial environment in which the electronic device is located and obtain at least one frame of environmental image; wherein, the spatial environment includes the desktop environment.

[0072] S302, identify at least one frame of environmental image to obtain the location and type information of items in the desktop environment.

[0073] S303, determine the functional information of the candidate items based on the type information of the candidate items.

[0074] In step S301, the acquisition module may include one or more cameras. The desktop robot can control the field of view of the acquisition module to face multiple different directions within the desktop environment, and capture multiple frames of environmental images in multiple directions through the acquisition module.

[0075] In step S302, the desktop robot can call a pre-built semantic segmentation model to perform item recognition processing on each frame of the acquired environmental image, so as to identify the type information of the candidate items displayed in each frame of the environmental image and determine the location information of each candidate item displayed.

[0076] The category information of the candidate item indicates what the candidate item is, combined with... Figure 2 In example (1), the category information of candidate item A01 can be identified from the environmental image as a water cup and the category information of candidate item A02 as a pen.

[0077] The location information of candidate items refers to their orientation within the spatial environment of the electronic device. This location information can be represented in different ways. Optionally, the desktop robot can establish a coordinate system based on its spatial environment, such as a Cartesian coordinate system with the corner of the desktop as the origin, and use the three-dimensional coordinates of the candidate item in this coordinate system as its location information.

[0078] Optionally, the desktop robot can also represent the position information of the candidate item using its orientation and position relative to the desktop robot. For example, the position information of the candidate item could be that it is located at the 3 o'clock position of the desktop robot, 40 centimeters away.

[0079] The location and category information of candidate items can be obtained by processing environmental images using image-based item detection and localization technologies in related technical fields. Specific processing methods are detailed in relevant technologies and will not be elaborated upon here. The algorithms and / or models used to obtain the location and category information can be deployed locally on the desktop robot or on a cloud-based server device, accessible to the desktop robot via a network.

[0080] In step S303, the method to obtain functional information based on category information can be to input the category information of the candidate item into a neural network model with reasoning ability (such as a large language model) to obtain the inherent function and applicable task scenario of the candidate item obtained by the model reasoning.

[0081] For example, if the candidate item is a pen, the model reasoning will determine that the pen's inherent functions include writing and its applicable task scenarios include meetings, taking notes, and attending classes. If the candidate item is a remote control, the model will determine that the remote control's inherent functions include controlling a projector and its applicable task scenarios include debugging a projector, presenting slides, and holding multi-person online meetings.

[0082] Next, a neural network model (such as a word embedding model) that can convert information into vectors can be invoked to process the inherent function and applicable task scenario of the candidate item, and an item function vector representing the inherent function and applicable task scenario of the candidate item can be obtained. This item function vector is then used as the functional information of the candidate item.

[0083] Optionally, item function vectors can be generated solely based on the task scenarios in which candidate items are used.

[0084] The models used above can be deployed on desktop robots or cloud server devices.

[0085] Optionally, considering that items of the same type can handle the same tasks and have the same functional information, after generating the functional information of a candidate item each time, the type information and functional information of this candidate item can be recorded in the database. When obtaining the type information of the same candidate item subsequently, the corresponding functional information can be directly read from the database as the functional information of the candidate item, without needing to generate it repeatedly.

[0086] As an example, when a candidate item A01 of the pen type is first identified in the spatial environment, the large language model is called to generate the functional information 1 corresponding to the pen. The functional information is used as the functional information of this candidate item, and the pen-functional information 1 is recorded in the database accordingly.

[0087] When step S101 is executed again and a candidate item of the pen type is identified in the environmental image, the functional information 1 corresponding to the pen is read from the database and used as the functional information of this candidate item.

[0088] The location and functional information of the candidate items obtained each time S101 is executed can be recorded in the desktop item database, which can be stored in the desktop robot's local storage space. Combined with... Figure 2 In example (1), after executing S101 once, the desktop item database can record the following information: pen - location information 1 - function information 1; stylus - location information 2 - function information 2; water cup - location information 3 - function information 3; remote control - location information 4 - function information 4; notebook - location information 5 - function information 5.

[0089] Optionally, a candidate item can correspond to one or more functional information. In the case of multiple functional information, each functional information represents an applicable task scenario for this candidate item. For example, a pen can correspond to three functional information, that is, three different item function vectors, denoted as functional information 1.1, functional information 1.2, and functional information 1.3, respectively. Functional information 1.1 represents that the pen can be used in a meeting scenario, functional information 1.2 represents that the pen can be used in a note-taking scenario, and functional information 1.3 represents that the pen can be used in a class scenario.

[0090] Before S102, the method in this embodiment further includes: obtaining user intent information.

[0091] User intent information can be obtained based on one or more of the user's behavior and user input. The form of user input is not limited; for example, user input can include one or more of voice input and text input.

[0092] Accordingly, obtaining user intent information specifically includes at least one of the following:

[0093] Method 1: Obtain user intent information based on user behavior;

[0094] Method 2: Obtain user intent information based on user input, which can be either voice or text input.

[0095] The third method involves obtaining user intent information based on user input when the user's behavior meets the triggering conditions.

[0096] In method one, the desktop robot can capture at least one frame of continuous user behavior image or a video of user behavior using its acquisition module. It then utilizes the recognition capabilities of a neural network model in related technologies to identify these user behavior images or videos, thereby obtaining the user behavior. For example, the obtained user behavior could be any of the following: standing up from a chair and stretching, picking up a notebook / coffee cup (or other item), or sitting down and moving closer to the table. The aforementioned user behaviors are merely examples; in actual applications, the identified user behaviors may not be limited to these.

[0097] After obtaining user behavior, based on user behavior and items affected by user behavior (such as the voice recorder mentioned above), combined with the user's historical behavior habits and the behavior habits of multiple user groups, the task that the user needs to process now can be inferred (denoted as the current task), and then user intent information can be generated based on the current task.

[0098] For example, if user A’s behavior is identified as “standing up from the seat and stretching”, based on user A’s historical behavior habits, user A is likely to go to the tea room after this behavior, so it is determined that user A’s current task is to get water from the tea room.

[0099] The user behavior of User A was identified as "sitting down and moving closer to the desktop". Based on User A's historical behavior habits, it is highly likely that User A will join an online meeting after this behavior. Therefore, the current task is determined to be an online meeting.

[0100] The user behavior of User A was identified as "picking up the notebook". Based on the user group behavior habits of multiple users, most users will pick up the notebook before going to the meeting room. Therefore, it was determined that the current task is an offline meeting.

[0101] Once the current task is determined, a task semantic vector representing the current task can be generated, which serves as user intent information.

[0102] In the second method, the ability of a neural network model to understand the meaning of speech or text can be used to parse the semantics of the user input, obtain the current task represented by the user input, and then generate a task semantic vector representing the current task. This task semantic vector serves as the user intent information.

[0103] As an example, when the voice input is "A meeting? I'm coming," it can be parsed to find that the current task is "in-person meeting"; when the text input is "Let's have a video meeting to discuss it in 5 minutes," it can be parsed to find that the current task is "online meeting."

[0104] Voice input can be acquired through the microphone of the desktop robot itself or through a microphone connected to the desktop robot. Text input can be acquired through the text input module of the desktop robot itself or through other electronic devices connected to the desktop robot. For example, if the desktop robot is connected to a computer on the desktop, the user can input and send messages in the computer's chat software, and the user's input message can be obtained as text input.

[0105] In method three, the user can pre-configure one or more trigger behaviors. When the user's behavior is identified as consistent with the pre-configured trigger behavior, or when the similarity between the identified user behavior and the trigger behavior exceeds a certain threshold, it can be determined that the user behavior meets the trigger condition. At this point, the desktop robot triggers the microphone to collect voice input and processes the collected voice input according to method two to obtain the corresponding user intent information. If the user behavior does not meet the trigger condition, the desktop robot does not trigger the microphone to collect voice input.

[0106] The advantage of the third method of application acquisition is that the desktop robot does not need to collect voice input through the microphone in real time, which helps to reduce power consumption.

[0107] Optionally, in order to improve the success rate of recommended items and accurately recommend the items needed by the user, after obtaining the current task, a neural network model with reasoning ability can be used to reason about the current task, determine the associated task semantics related to the current task, and then generate a task semantic vector as user intent information based on the current task and associated task semantics.

[0108] For example, if the current task is "offline meeting", the inferred related task semantics can include meeting, recording and writing. Subsequently, user intent information can be generated based on "offline meeting" and "meeting, recording and writing".

[0109] Optionally, if the user input includes information related to items, the task item requirement can be determined based on this information, and user intent information can be generated based on the current task, associated task semantics, and task item requirement. For example, when receiving the voice input "You have a meeting? Come right away, where's my pen?", it can be determined that the current task is "offline meeting", the associated task semantics are "meeting, recording, and writing", and the task item requirement is "finding a pen", and user intent information can be generated based on these three factors.

[0110] Among the above methods, neural network models with semantic understanding and feature vector extraction capabilities can be used to process the current task and obtain user intent information representing the current task.

[0111] After obtaining user intent information, the matching degree between the user intent information and each function information of each candidate item obtained most recently can be calculated. The calculation method is not limited. As an example, the method of calculating vector similarity in related technologies can be used to calculate the similarity between user intent information and function information. The calculation result is used as the matching degree between the two.

[0112] Then, based on the matching degree of each functional information, recommended items can be determined from the most recently obtained candidate items. For example, the candidate item corresponding to the functional information with the highest matching degree can be determined as the recommended item; the candidate item corresponding to the functional information with a matching degree greater than a preset matching threshold can be determined as the recommended item (in this case, there can be multiple recommended items); and the candidate item corresponding to the functional information with a matching degree greater than the matching threshold and the highest matching degree can be determined as the recommended item.

[0113] As an example, after calculation, it was found that only the matching degree between functional information 1 and user intent information was greater than the matching threshold, while the matching degree of other information was less than the matching threshold. Therefore, the pen corresponding to functional information 1 was determined as the recommended item.

[0114] Optionally, a prompting action to indicate the location of the recommended item is performed based on the item's location information, including at least one of the following:

[0115] The first type of prompt operation could be to control the projection module of the electronic device to project onto the location of the recommended item based on the item's location information;

[0116] The second type of prompt operation can be to output prompt information based on the location information of the recommended item. The prompt information can be voice information, text information, or image information.

[0117] In the first type of prompting operation, the desktop robot can read the location information of the recommended item, such as reading the location information of the most recently acquired and written recommended item from the aforementioned desktop item database. Based on the location information, the desktop robot determines the direction and distance of the recommended item relative to itself, controls its projection module to rotate to face the direction of the recommended item, and then controls the projection module to emit a projection beam in that direction and distance, thereby projecting a highlighted area at the location of the recommended item to the user.

[0118] In the second type of prompting operation, the desktop robot can read the location information of the recommended item, generate text information describing the location of the recommended item based on the location information, and display this text information. For example, the displayed text information could be "The remote control is at the far right of the desktop," or "The water cup is to the left of the desktop robot." The above text information can describe the position of the recommended item relative to its spatial environment (such as the desktop environment), the position of the recommended item relative to the desktop robot, or the orientation of the recommended item relative to other items on the desktop.

[0119] In the second type of prompting operation, the desktop robot can also use text-to-speech technology to convert the above text information into speech information and play the speech information through a speaker.

[0120] In the second type of prompting operation, the desktop robot can also obtain at least one frame of environmental image of the surrounding space through the acquisition module, highlight the recommended item in the environmental image based on the location information of the recommended item, such as highlighting it or displaying a prompt box surrounding the recommended item, and finally display the environmental image with the recommended item highlighted as the above image information output.

[0121] The above text and image information can be displayed in any way, such as on a display screen or projected through a projection module.

[0122] Combination Figure 2In example (2), assuming the recommended item is determined to be a pen, the desktop robot reads the location information of the most recently obtained pen and, based on this location information, emits a projection beam in the direction and distance of the pen, such as... Figure 2 As shown by the dotted line in (2), a highlighted area is formed at the location of the pen.

[0123] Alternatively, you can follow the second prompt method, such as displaying the text message "The pen is on the right side of the desktop robot, next to the water cup," or playing the voice message "The pen is in the center of the desktop."

[0124] Optionally, after determining the recommended items, the desktop robot can also perform one or more of the following task assistance operations to help the user handle the current task more smoothly, including at least one of the following:

[0125] Task Assist Operation 1: Trigger electronic devices and / or other electronic devices to launch the target application. The target application can handle the same tasks as the recommended items.

[0126] Task Assistance Operation 2: Switch the electronic device from the current operating mode to the target operating mode. The electronic device in the target operating mode can handle the same tasks as the recommended items.

[0127] Task assistance operation three: Display and output pending task prompts. The pending task prompts indicate the next task that may need to be processed after the current task.

[0128] In Task Assistance Operation 1, electronic devices and / or other electronic devices refer to the executor of the item recommendation method and / or terminal electronic devices that are communicatively connected to this executor, such as desktop robots and / or mobile phones and tablets that are communicatively connected to desktop robots.

[0129] The tasks that each application can handle can be determined based on the user's usage habits when handling various tasks. For example, when handling offline meeting tasks, users often open the recording application on their mobile phones to record the meeting, so it can be determined that the recording application can handle offline meeting tasks; when handling online meeting tasks, users always open the video conferencing application and the slide presentation application on their computer, so it can be determined that the video conferencing application and the slide presentation application can handle online meeting tasks.

[0130] Based on this, the desktop robot can pre-generate a feature vector for each application according to the tasks that the application can handle, and use this feature vector as the application task information for that application.

[0131] Then, after each recommended item is determined, the desktop robot can calculate the matching degree between the functional information of the recommended item and the application task information of the applications on each electronic device, or it can calculate the matching degree between the user intent information and the application task information of the applications on each electronic device. The application with the highest matching degree with the functional information of the recommended item is selected as the target application, and / or the application with the highest matching degree with the user intent information is selected as the target application. The method for calculating the matching degree can be found in the method for calculating vector similarity.

[0132] The advantage of the task assistance operation is that it can pre-launch the target applications that the user needs to use when handling the current task, thus avoiding the situation where the user forgets to open the application in time while handling the task.

[0133] In Task Assistance Operation 2, for any operating mode, the tasks that the electronic device in this operating mode can handle can be determined as follows:

[0134] The system analyzes whether the electronic device has been in this operating mode when a user has previously handled a certain type of task. If the electronic device is always in this operating mode or is in this operating mode most of the time when handling a certain type of task, then it is determined that the electronic device in this operating mode is capable of handling this type of task.

[0135] As an example, when handling offline meeting tasks, users always set their phones to silent mode, thus confirming that phones in silent mode can handle offline meeting tasks; when handling online meeting tasks, users always set their computers to high-performance mode, thus confirming that computers in high-performance mode can handle online meeting tasks.

[0136] For each operating mode, the desktop robot can generate a corresponding feature vector based on the tasks that the electronic device in this operating mode can handle, and use this vector as the mode task information corresponding to this operating mode.

[0137] Therefore, after each recommended item is determined, the desktop robot can calculate the matching degree between the functional information of the recommended item and the mode task information of the operating modes of each electronic device, or it can calculate the matching degree between the user's intent information and the mode task information of the operating modes of each electronic device. The operating mode with the highest matching degree with the functional information of the recommended item is selected as the target operating mode, and / or the operating mode with the highest matching degree with the user's intent information is selected as the target operating mode. The method for calculating the matching degree can be found in the method for calculating vector similarity.

[0138] The advantage of the second task assistance operation is that it can pre-set the electronic device to the target operating mode that matches the current task that the user needs to handle, thus avoiding the situation where the user forgets to set the operating mode during the task handling process.

[0139] In Task Assistance Operation 3, the desktop robot can determine the next task that may need to be handled after the current task is completed, based on the user's pre-configured recent task plan and / or the user's historical task handling habits.

[0140] In one example, the next task can be determined based on a pre-configured task schedule. The user has pre-configured their task schedule for the day, which includes attending an offline meeting at 10:00 and writing a report at 11:00. After the desktop robot obtains the user's intent to attend the offline meeting at 10:00 and determines recommended items based on the user's intent, writing the report can be determined as the next task, and a to-do task prompt message about writing the report will be displayed and output.

[0141] In another example, the next task can be determined based on the user's historical task processing habits. Previously, after each online meeting, the user would write meeting minutes. Based on this historical task processing habit, after obtaining information indicating the user's intent to participate in the online meeting, the desktop robot can determine that writing meeting minutes is the next task and display a to-do message regarding writing meeting minutes.

[0142] Task reminders can be displayed on a screen or projector.

[0143] Optionally, determining recommended items from candidate items based on the obtained user intent information and the functional information of candidate items may include:

[0144] Obtain user usage history information, including items used by the user;

[0145] Recommended items are determined from the candidate items based on usage records, user intent information, and functional information of the candidate items.

[0146] Usage log information can include items used by the user within a preset time period up to the current moment (e.g., 1 week, 15 days, etc., set as needed), and can specifically include information on the types of items used by the user each time.

[0147] As an example, usage log information may include the following records: using a pen, using a remote control, using a pen, using a stylus, using a pen.

[0148] Desktop robots can determine which items a user has used to obtain the aforementioned usage record information by using the following methods:

[0149] The desktop robot periodically obtains information on the types of candidate items within its environment based on a certain time cycle. If information on the type of a certain candidate item X is missing in the current iteration compared to the previous iteration, and this information is obtained again after at least a preset time (e.g., 5 minutes, 3 minutes, etc., set as needed), then it is determined that this type of candidate item has been used once, and a corresponding record is added to the usage log. The method for obtaining the type information of candidate items can be found in the previous text.

[0150] As an example, the first type of information obtained includes a pen, indicating that a pen is placed on the table. The second type of information obtained does not include a pen, indicating that the pen on the table was taken away by the user. After 6 minutes, a pen is found again in the obtained type of information, indicating that the pen was taken away by the user, used, and then put back on the table. Thus, it can be determined that the pen was used once, and a new record "Pen Used" can be added to the usage record information.

[0151] One way to determine recommended items by combining recorded information is as follows:

[0152] After determining the matching degree based on user intent information and candidate item functional information, if only one candidate item has a matching degree that meets the recommendation criteria (such as being greater than the matching degree threshold), this candidate item is directly identified as the recommended item. If two or more candidate items have matching degrees that are greater than the matching degree threshold, the candidate item with the most recorded usage frequency (i.e., the most usage records) among the multiple candidate items with matching degrees greater than the matching degree threshold can be identified as the recommended item by combining usage record information.

[0153] As an example, suppose the user intent information indicates that the current task is an offline meeting task, and the candidate items in the current desktop environment are a remote control, a pen, and a water cup. Only the pen has a matching degree greater than the matching degree threshold, so the pen is determined to be the recommended item.

[0154] If the candidate items in the current desktop environment are a remote control, a pen, a voice recorder, and a water cup, and considering that writing notes or recording may be necessary during meetings, the matching degree of the pen and the voice recorder is greater than the matching degree threshold, then the usage record information is queried. It is found that there are 5 usage records of the pen and 1 usage record of the voice recorder. Therefore, it is determined that the pen, which is used the most, is the recommended item.

[0155] The advantage of determining recommended items in the above way is that it can combine users' preferences for using different types of items, and when there are multiple candidate items that meet the recommendation criteria, it can more accurately determine the recommended items that meet the user's actual needs.

[0156] Optionally, recommended items can be determined from the candidate items based on the obtained user intent information and the functional information of the candidate items, including:

[0157] If at least one candidate item meets the recommendation criteria, the candidate item that meets the recommendation criteria is determined as the recommended item. The matching degree of the candidate item is determined based on the user intent information and the functional information of the candidate item.

[0158] If the matching degree of each candidate item does not meet the recommendation conditions, obtain the location information and functional information of the updated candidate items.

[0159] Recommended items are determined from the updated candidate items based on user intent information and the functional information of the updated candidate items.

[0160] For the method of obtaining the location information and functional information of the updated candidate items, please refer to the method of obtaining the location information and functional information of the candidate items in step S101.

[0161] Combination Figure 3 If the matching degree of each candidate item does not meet the recommendation criteria, the desktop robot can call its own acquisition module to scan and obtain at least one frame of the desktop environment image. Then, it sequentially executes S302 and S303 on the scanned environment image. The position information obtained at this time is the updated position information of the candidate items, and the functional information obtained is the updated functional information of the candidate items. In other words, if the matching degree of each candidate item does not meet the recommendation criteria, the desktop robot will proceed according to... Figure 3 The method involves rescanning the items in the desktop environment and re-determining the location and function information of the scanned items. The re-scanned items are then identified as the updated candidate items.

[0162] The method for determining recommended items based on user intent information and updated candidate item functional information can be found in the foregoing embodiments of the method for determining recommended items based on user intent information and candidate item functional information, and will not be repeated here.

[0163] As an example, suppose the user intent information indicates that the current task is an offline meeting task, and the most recently obtained candidate items are a remote control and a water cup, neither of which meets the recommendation criteria in terms of their functional information.

[0164] Therefore, the desktop robot rescans the items on the desktop environment to obtain the updated functional information and location information of the candidate items. The functional information obtained at this time includes the functional information of the remote control, the pen, and the water cup. Based on the matching degree, it is determined that the pen meets the recommendation conditions, and thus the pen is determined to be the recommended item.

[0165] After the desktop robot obtains the location and function information of the candidate items, the candidate items in the spatial environment may change, resulting in incomplete information (including location and function information) of the candidate items obtained by the desktop robot. For example, at time T1, the desktop robot obtains information about the candidate items in the desktop environment once, and then at time T2, the user places a pen on the desktop. This results in incomplete information about the candidate items on the desktop obtained by the desktop robot, and thus it cannot determine the recommended items in time.

[0166] By determining recommended items using the above method, the latest information on items in the current spatial environment can be obtained in a timely manner when the above situation occurs, thereby promptly identifying recommended items among the items placed in the current spatial environment and improving the success rate of determining recommended items when applying the method of this embodiment.

[0167] Optionally, the desktop robot can trigger the execution of steps to obtain the location and functional information of candidate items based on any one or more of the following triggering methods:

[0168] Triggering method one: In response to obtaining at least one of the user's user behavior and user input, the location information and functional information of the candidate item are obtained, and at least one of the user behavior and user input is used to determine the user's intent information;

[0169] Triggering method two: Obtain the location and functional information of candidate items based on a time period;

[0170] Triggering method three: In response to any change in the quantity, type, posture, or position of items in the spatial environment where the electronic device is located, the location and functional information of the candidate items are obtained.

[0171] Triggering method four involves responding to the user's trigger command by obtaining the location and functional information of the candidate items.

[0172] Triggering method one can also be changed to execute the step of obtaining the location information and function information of the candidate item once each time user intent information is obtained based on at least one of the user's user behavior and user input.

[0173] In the case of application triggering method one, after the desktop robot is started, it can temporarily not scan the items in the surrounding space environment, while detecting whether there is user behavior and / or user input in real time. Whenever at least one of user behavior and user input is obtained, and the user intent information is determined based on the user behavior and / or user input, the desktop robot scans the items in the surrounding space environment once in the manner of step S101 to obtain the location information and functional information of the candidate items, and then executes steps S102 and S103 based on the location information and functional information of the candidate items obtained in this scan.

[0174] In other words, when applying triggering method one, the implementation process of the item recommendation method in this embodiment may include... Figure 4 The steps are shown.

[0175] S401, User behavior and / or user input were detected.

[0176] S402, determine user intent information based on user behavior and / or user input.

[0177] S403, scan the surrounding environment once to obtain the location and functional information of the candidate items.

[0178] S404, determine recommended items based on the matching degree of functional information.

[0179] S405, execute the prompted operation.

[0180] The implementation methods of S401 and S402 can be found in the previous section on obtaining user intent information. The implementation method of S403 is found in S101, the implementation method of S404 is found in S102, and the implementation method of S405 is found in S103. Specifically, S405 performs a prompting operation based on the location information obtained during the most recent execution of S403.

[0181] When any one of the application triggering methods two to four is applied, the desktop robot can execute step S101 once after startup to record the location information and function information of the obtained candidate items in the desktop item database.

[0182] Subsequently, if the conditions corresponding to any of the triggering methods two to four are not met, step S101 will not be executed. If the conditions corresponding to any of the triggering methods two to four are met, step S101 will be executed once, and the location information and function information of the candidate items obtained after this execution will replace the information previously recorded in the desktop item database.

[0183] Each time the desktop robot obtains user intent information based on user behavior and / or user input, it executes the aforementioned item recommendation method according to the location and function information recorded in the desktop item database. This eliminates the need to scan the surrounding environment every time user behavior and / or user input is obtained. This saves power consumption from multiple scans and also saves time in executing S101, shortening the time from obtaining user behavior and / or user input to executing the prompting operation, thus enabling faster execution of the prompting operation.

[0184] The desktop robot can also execute step S101 to obtain the updated location information and function information of the candidate items when it obtains user intent information and the matching degree between the function information and user intent information of each candidate item does not meet the recommendation conditions. Then, on the one hand, it redetermines the recommended items based on this information, and on the other hand, it records this information as the location information and function information of the candidate items in the desktop item database for use when user intent information is obtained next time.

[0185] In other words, when any one of triggering methods two to four is applied, the item recommendation method in this embodiment may include Figure 5 The steps are shown.

[0186] S501, User behavior and / or user input were detected.

[0187] S502, determine user intent information based on user behavior and / or user input.

[0188] S503 determines the matching degree based on the functional information recorded in the desktop item database, and then determines recommended items based on the matching degree.

[0189] S504, based on the location information of recommended items recorded in the desktop item database, perform the prompt operation.

[0190] S505 determines in real time whether an update is needed.

[0191] If any one of the trigger conditions two to four is met, or if the matching degree between the functional information and user intent information of each candidate item does not meet the recommendation conditions, then it is determined that an update is needed; otherwise, it is determined that no update is needed.

[0192] If an update is needed, execute S506; otherwise, leave the current desktop item database unchanged.

[0193] S506 scans the surrounding environment once to obtain the location and functional information of candidate items, and stores the obtained information in the desktop item database.

[0194] The implementation methods of S501 and S502 can be found in the previous section on obtaining user intent information; the implementation method of S503 can be found in S102; the implementation method of S504 can be found in S103; and the implementation method of S506 can be found in S101.

[0195] The time period for triggering method two can be set as needed without limitation. For example, the time period can be set to once every 3 minutes, once every 10 minutes, etc.

[0196] In triggering mode three, the desktop robot can periodically detect one or more of the quantity, type, posture and position of objects in the space environment based on the environmental images collected by the acquisition module, and record the results of each detection. If the result of a certain detection is different from the previous one, it is determined that any of the quantity, type, posture and position of objects in the space environment where the electronic device is located has changed, and thus triggering the execution of step S101.

[0197] The posture of the items includes, but is not limited to: the direction of the handle of the water cup, the open and closed state of the notebook, the open and closed state of the foldable screen device, and the tilt angle of the pen.

[0198] In triggering method four, the form of the user's trigger command is not limited. For example, it can be a fixed voice command, such as "scan the items on the desktop", or the user can press a specific button on the desktop robot or make a specific gesture. It can also be any voice input that can express the intention to find an item, such as "help me find something" or "check the items on the desktop".

[0199] The following examples illustrate the implementation process of the item recommendation method in this embodiment in real-world application scenarios.

[0200] Example 1: The desktop robot collects the user's voice input, "You have a meeting? I'll be right there. Where's my pen?". Based on the user's voice input, it determines: the current task is an offline meeting; the required item is a pen; and the associated task semantics include meeting, note-taking, and writing.

[0201] Generate user intent information indicating that the user needs to handle an offline meeting task (i.e., attend an offline meeting) based on the current task, task item requirements, and related task semantics;

[0202] Read the desktop item database to obtain the functional information of multiple candidate items recorded therein, such as the functional information of a pen, a stylus, and a marker. Match the user intent information with the functional information of multiple candidate items recorded in the desktop item database to obtain the matching degree corresponding to each candidate item.

[0203] Based on the user intent information indicating the offline meeting task, the pen was determined to have the highest matching degree among all candidate items. Therefore, the desktop robot controlled the projection module to project the image onto the pen. Figure 2 As shown in (2), and a prompt message describing the position of the pen is displayed on the projection interface.

[0204] Example 2: The desktop robot collects the user's voice input "Is there something wrong with the projector? Let me take a look." Based on the user's voice input, it generates user intent information, which indicates that the current task is to check the projector.

[0205] The desktop robot scans the desktop environment through the acquisition module to obtain an environmental image, identifies candidate items placed on the desktop from the environmental image, and obtains the location and functional information of these candidate items. The functional information includes the functional information of the remote control, the functional information of the water cup, and the functional information of the headphones.

[0206] The remote control's functional information indicates that it can handle tasks related to the projector. Therefore, after calculating the matching degree with the user's intent information, the remote control is determined to be the recommended item, and the projection module is controlled to project onto the remote control based on the previously obtained location information of the remote control.

[0207] Example 3: The desktop robot recognizes the user behavior of "standing up from the seat and stretching" based on the image captured by the acquisition module. Based on this user behavior, it generates user intention information indicating that the user is going to the tea room to get water. It calculates the matching degree between this user intention information and the functional information of the candidate items on the scanned desktop. Based on the matching degree, it determines that the water cup is the recommended item, and then controls the projection module to project onto the water cup.

[0208] In some optional embodiments, to avoid accidental triggering that could affect the user's normal use of items, the desktop robot may only execute the item recommendation method provided in any embodiment when it receives voice input containing a specific wake word. For example, if the received voice input is "Xiao X, I'm going to a meeting," the item recommendation method is executed to obtain user intent information representing an offline meeting task and to perform corresponding prompts. If the received voice input is "I'm going to a meeting," the item recommendation method is not executed, where "Xiao X" is a predefined wake word.

[0209] This application also provides an electronic device for performing the item recommendation method of any of the foregoing embodiments. This electronic device may be the desktop robot of the foregoing embodiments, or other electronic devices that can achieve the same function.

[0210] See Figure 6 This is a schematic diagram of the structure of an electronic device, which may include a data acquisition module 601, a memory 602 and a processor 603.

[0211] The acquisition module 601 is used to detect objects in the spatial environment where the electronic device is located;

[0212] Memory 602 is used to store computer programs;

[0213] Processor 603 is used to execute computer programs to perform:

[0214] Obtain the location and functional information of candidate items. The functional information at least represents the task that the corresponding candidate item can handle. Candidate items include items in the spatial environment where the electronic device is located.

[0215] Recommended items are determined from the candidate items based on the obtained user intent information and the functional information of the candidate items. The user intent information at least represents the task that the user needs to process at present.

[0216] Execute a prompt action to indicate the location of the recommended item based on the item's location information.

[0217] In some optional embodiments, the electronic device provided in this embodiment may further include Figure 7 The projection module 701 shown is used to project onto any position.

[0218] When the projection module 701 is included, the processor 603, when performing the prompting operation, specifically controls the projection module to project onto the location of the recommended item based on the location information of the recommended item.

[0219] Optionally, the electronic device provided in this embodiment may further include an output module for outputting prompt information determined by the processor based on the location information of the recommended items. The prompt information may be voice information, text information, or image information.

[0220] When the output module is included, the processor 603 can specifically control the output module to output prompt information when performing a prompting operation.

[0221] The working principle of the above electronic devices can be found in the relevant steps of the item recommendation method provided in any of the foregoing embodiments, and will not be repeated here.

[0222] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0223] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0224] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0225] Finally, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for recommending items, applied to electronic devices, comprising: Obtain the location and functional information of candidate items, wherein the functional information at least characterizes the task that the corresponding candidate item can handle, and the candidate items include items within the spatial environment where the electronic device is located; Recommended items are determined from the candidate items based on the obtained user intent information and the functional information of the candidate items, wherein the user intent information at least represents the task that the user currently needs to process; Based on the location information of the recommended item, a prompting operation is performed to indicate the location of the recommended item.

2. The method according to claim 1, wherein obtaining the location information and functional information of the candidate items includes: Scan the spatial environment in which the electronic device is located to obtain at least one frame of environmental image; wherein, the spatial environment includes the desktop environment; The location and type information of items within the desktop environment are obtained by identifying at least one frame of the environmental image. The functional information of the candidate items is determined based on the type information of the candidate items.

3. The method according to claim 1, wherein the step of performing a prompting operation to indicate the location of the recommended item based on the location information of the recommended item includes at least one of the following: Based on the location information of the recommended item, the projection module of the electronic device is controlled to project onto the location of the item; Output prompt information determined based on the location information of the recommended items. The prompt information may be voice information, text information, or image information.

4. The method according to claim 1, further comprising: Obtaining the user's intent information specifically includes at least one of the following: The user intent information is obtained based on the user's behavior. The user intent information is obtained based on the user's input, which can be either voice input or text input. When a user's behavior meets the triggering conditions, the user intent information is obtained based on the user's input.

5. The method according to claim 1, wherein determining the recommended item from the candidate items based on the obtained user intent information and the functional information of the candidate items comprises: Obtain the user's usage record information, which includes items used by the user; Based on the usage record information, the user intent information, and the functional information of the candidate items, a recommended item is determined from the candidate items.

6. The method of claim 1, further comprising at least one of the following: Trigger the electronic device and / or other electronic devices to launch the target application, the target application being able to handle the same tasks as the recommended item; Switch the electronic device from its current operating mode to a target operating mode. The electronic device in the target operating mode is capable of handling the same tasks as the recommended item.

7. The method according to claim 1, wherein determining the recommended item from the candidate items based on the obtained user intent information and the functional information of the candidate items comprises: If at least one candidate item's matching degree meets the recommendation criteria, the candidate item whose matching degree meets the recommendation criteria is determined as a recommended item. The matching degree of the candidate item is determined based on the user intent information and the functional information of the candidate item. If the matching degree of each candidate item does not meet the recommendation conditions, obtain the location information and functional information of the updated candidate items. Recommended items are determined from the updated candidate items based on the user intent information and the functional information of the updated candidate items.

8. The method according to claim 1, wherein obtaining the location information and functional information of the candidate items includes at least one of the following: In response to obtaining at least one of the user's user behavior and user input, the location information and functional information of the candidate item are obtained, wherein at least one of the user behavior and user input is used to determine the user intent information; The location and functional information of candidate items are obtained based on the time period; In response to any change in the quantity, type, orientation, or position of items in the spatial environment where the electronic device is located, the location and functional information of the candidate items are obtained. In response to the user's trigger command, obtain the location and functional information of the candidate items.

9. An electronic device, comprising a data acquisition module, a memory, and a processor; The acquisition module is used to detect items in the spatial environment where the electronic device is located; The memory is used to store computer programs; The processor is used to execute the computer program to perform: Obtain the location and functional information of candidate items, wherein the functional information at least characterizes the task that the corresponding candidate item can handle, and the candidate items include items within the spatial environment where the electronic device is located; Recommended items are determined from the candidate items based on the obtained user intent information and the functional information of the candidate items, wherein the user intent information at least represents the task that the user currently needs to process; Based on the location information of the recommended item, a prompting operation is performed to indicate the location of the recommended item.

10. The device according to claim 9, further comprising: The projection module is used to project onto any location; The processor is specifically used to control the projection module to project onto the location of the recommended item based on the location information of the recommended item; Or, it may also include: The output module is used to output the prompt information determined by the processor based on the location information of the recommended item. The prompt information may be voice information, text information, or image information.

Images