Data processing method and wearable device

Abstract

The invention discloses a data processing method and wearable equipment, and the method comprises the steps: obtaining a plurality of first sensing data through a plurality of sensors disposed on the wearable equipment; performing data perception on the first sensing data and attention recognition of a wearing user by using an intelligent agent deployed on the wearable device so as to determine a target object; obtaining multiple items of second sensing data according to the target object; the second sensing data is related to the target object; and executing at least one target instruction according to the first sensing data and the second sensing data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a data processing method and a wearable device. BACKGROUND

[0002] At present, the sensor directly collects sensing data on the wearable device and provides intelligent services for users accordingly. SUMMARY

[0003] Therefore, the present application provides a data processing method and a wearable device, as follows.

[0004] A data processing method comprises the following steps.

[0005] A plurality of first sensing data are obtained by a plurality of sensors deployed on a wearable device.

[0006] An agent deployed on the wearable device is used to perform data sensing on the first sensing data and to identify the attention of a wearer, so as to determine a target object.

[0007] According to the target object, a plurality of second sensing data are obtained; the second sensing data are related to the target object.

[0008] At least one target instruction is executed according to the first sensing data and the second sensing data.

[0009] In the above method, preferably, according to the target object, a plurality of second sensing data are obtained, comprising the following steps.

[0010] An object type to which the target object belongs is determined; the object type comprises the wearer himself / herself, a real environment in which the wearer is located, and a virtual environment watched by the wearer.

[0011] According to the object type, a plurality of second sensing data are obtained in a target manner.

[0012] The target manner corresponds to the object type.

[0013] In the above method, preferably, the target manner comprises one of the following.

[0014] The sensor is controlled to collect sensing data for the target object, so as to obtain a plurality of second sensing data.

[0015] Second sensing data related to the target object are extracted from the first sensing data.

[0016] In the above method, preferably, the sensor is controlled to collect sensing data for the target object, so as to obtain a plurality of second sensing data, comprising the following steps.

[0017] control the sensor to collect the sensing data meeting the correlation relationship with the target object with a first collection parameter, and control the sensor to collect the sensing data meeting the interference condition with a second collection parameter, to obtain a plurality of second sensing data;

[0018] wherein the data collection intensity of the sensor under the first collection parameter is higher than that under the second collection parameter.

[0019] The above method, preferably, in the case that the object type is the wearer himself, the second sensing data includes one of the following:

[0020] environmental data within the blind area of the wearer, the target instruction corresponding to the environmental data being used to output first prompt information, the first prompt information being used to prompt a key object within the blind area;

[0021] vital sign data of the wearer, the target instruction corresponding to the vital sign data being used to output second prompt information, the second prompt information being used to prompt a physical state of the wearer.

[0022] The above method, preferably, after the first sensing data is subjected to data perception and attention recognition of the wearer by an agent deployed on the wearable device to determine a target object, the method further comprises:

[0023] in the case that the target object is a real environment in which the wearer is located, according to user information of the wearer, filtering target information associated with the target object in a knowledge base;

[0024] outputting the target information.

[0025] The above method, preferably, after the first sensing data is subjected to data perception and attention recognition of the wearer by an agent deployed on the wearable device to determine a target object, the method further comprises:

[0026] in the case that the target object is a virtual environment viewed by the wearer, according to content output by the virtual environment and input operation of the wearer on the virtual environment, determining an operation intention of the wearer on the virtual environment;

[0027] outputting action prompt information according to the operation intention, the action prompt information being used to prompt the wearer to perform an action on the virtual environment.

[0028] The above method, preferably, the method further comprises:

[0029] processing the first sensing data by using the agent to determine an environment type of an environment where the wearable device is located;

[0030] in a case where the environment type matches a preset alarm type, outputting risk prompt information.

[0031] A wearable device comprises:

[0032] a plurality of sensors configured to collect a plurality of first sensing data;

[0033] a processor configured to deploy an agent, the agent being configured to perform data sensing on the first sensing data by using the agent deployed on the wearable device and to recognize attention of a wearer to determine a target object; obtain a plurality of second sensing data according to the target object, the second sensing data being related to the target object; and execute at least one target instruction according to the first sensing data and the second sensing data.

[0034] In the above wearable device, preferably, the sensors comprise at least two of the following:

[0035] an image sensor, a sound sensor, a motion sensor, a biological sensor, and an environment sensor;

[0036] The image sensor is configured to collect image data.

[0037] The sound sensor is configured to collect sound data.

[0038] The motion sensor is configured to collect motion data of the wearable device and / or body posture data of the wearer.

[0039] The biological sensor is configured to collect vital sign data of the wearer.

[0040] The environment sensor is configured to collect environment data of an environment where the wearable device is located.

[0041] From the above technical solution can be seen, the data processing method and wearable device disclosed in the present application, after collecting the first sensing data through the sensor on the wearable device, the first sensing data is perceived by the agent on the wearable device, and the attention of the wearer is recognized, and then the target object attracting the attention of the wearer is determined, at this time, according to the target object, the second sensing data related to the target object is obtained again, and according to the first sensing data and the second sensing data of the target object attracting the attention of the wearer, the target instruction realizing the corresponding function is executed. It can be seen that in the present application, the target instruction is executed in combination with the sensing data of the object attracting the attention of the wearer, which can make the realized function more matched with the matter concerned by the wearer, thereby more satisfying the user's use demand, and further enabling the wearable device to provide more accurate intelligent service for the user, and improving the user's experience of using the wearable device. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 A flowchart of a data processing method provided by the embodiment of the present application;

[0044] Figure 2 An example diagram of executing a target instruction for a target object in the embodiment of the present application;

[0045] Figure 3 Another example diagram of executing a target instruction for a target object in the embodiment of the present application;

[0046] Figure 4 A partial flowchart of a data processing method provided by the embodiment of the present application;

[0047] Figure 5 、 Figure 6 、 Figure 7 、 Figure 8 、 Figure 9 、 Figure 10 Another example diagram of executing a target instruction for a target object in the embodiment of the present application;

[0048] Figure 11 A structural schematic diagram of a data processing device provided by the embodiment of the present application;

[0049] Figure 12 Another structural schematic diagram of a data processing device provided by the embodiment of the present application;

[0050] Figure 13 A structural schematic diagram of a wearable device provided by an embodiment of the present application is shown in the figure.

[0051] Figure 14 A module structural diagram of the present application applicable to a smart earphone is shown in the figure.

[0052] Figure 15 An implementation framework diagram of the present application applicable to a smart earphone is shown in the figure.

[0053] Figure 16 A data processing flowchart of the present application applicable to a smart earphone is shown in the figure.

[0054] Figure 17 An integrated design schematic diagram of a smart earphone in a scenario of the present application applicable to a smart earphone is shown in the figure.

[0055] Figure 18 An example diagram of a smart earphone and an earphone box in a split design in a scenario of the present application applicable to a smart earphone is shown in the figure.

[0056] Figure 19 An architecture composition and processing flow schematic diagram of the present application applicable to a smart earphone is shown in the figure. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0058] Reference Figure 1 An implementation flowchart of a data processing method provided by an embodiment of the present application is shown in the figure. The method can be applicable to a wearable device, such as a smart glass, a smart earphone, or a smart watch, etc. The technical solution in the present embodiment is mainly used to improve the user experience of the wearable device.

[0059] Specifically, the method in the present embodiment can include the following steps:

[0060] Step 101: Obtain a plurality of first sensing data through a plurality of sensors deployed on the wearable device.

[0061] The sensors deployed on the wearable device can include at least two of an image sensor, a sound sensor, a motion sensor, a biological sensor, and an environmental sensor, etc.

[0062] For example, the image sensor can include a camera for collecting image data such as a 360-degree panoramic overhead image; the sound sensor can be a microphone for collecting sound data; the motion sensor can include an inertial measurement unit (IMU), a gravity sensor (Gsensor), etc., for collecting motion data of the wearable device and / or body posture data of the wearing user; the biological sensor (bio) can include an electroencephalogram (EEG) sensor, an electrocardiogram sensor, a blood pressure sensor, etc., for collecting vital sign data of the wearing user, such as brain waves, heartbeats, blood pressure, etc.; the environmental sensor can include a millimeter wave radar, a temperature sensor, a humidity sensor, etc., for collecting environmental data of the environment where the wearable device is located, such as radar detection data, temperature data, humidity data, etc. of the environment. Based on this, the first sensor data can include at least two of the above image data, sound data, motion data, body posture data, vital sign data, and environmental data.

[0063] Step 102: data sensing of the first sensor data and attention recognition of the wearing user by the agent deployed on the wearable device to determine the target object.

[0064] The target object is an object that attracts the attention of the wearing user recognized by the agent, and the target object can include the wearing user himself / herself, a person or object in the surrounding environment, a person or object in a virtual environment (projected picture or display screen), etc.

[0065] It should be noted that the agent on the wearable device can be an intelligent model capable of data reasoning. Specifically, the data reasoning of the agent includes data sensing processing of the first sensor data and attention recognition of the wearing user based on the sensed results, so that the agent can output the matter of interest of the wearing user, i.e., the target object.

[0066] Specifically, the agent can be a pre-trained machine learning model, which can collect, organize, calculate, analyze, and reason data of input images, sounds, texts, etc. to obtain corresponding reasoning results. Based on this, the agent can perform data reasoning on the first sensor data to recognize the target object that attracts the attention of the wearing user.

[0067] It should be noted that the agent can perform data reasoning, such as attention recognition and behavior intention recognition, in cooperation with a large model in the cloud.

[0068] Step 103: obtaining a plurality of second sensor data according to the target object, the second sensor data being related to the target object.

[0069] The second sensing data can be obtained by a plurality of sensors deployed on the wearable device, or the second sensing data can be obtained by sensors on other devices connected to the wearable device, and then the second sensing data is sent to the wearable device by the other devices.

[0070] For example, taking the target object as a person in the surrounding environment as an example, as shown in Figure 2 After the smart glasses carried by the user collect image data in the surrounding environment and body posture data and sound data of the user himself through a plurality of sensors, the agent in the smart glasses reasons out that the user is interested in a certain person through the whispering sound data "who is this person", the agent identifies the "certain person" in the surrounding environment within the range seen by the user through the image data, and based on this, the agent determines that the target object attracting the user's attention is the "certain person", and thus the agent uses the sensors to collect sensing data for the target object, such as focusing on collecting image data of the "certain person", thereby obtaining the second sensing data.

[0071] For example, taking the target object as a person in the surrounding environment as an example, as shown in Figure 3 After the smart glasses carried by the user collect image data in the surrounding environment and body posture data and sound data of the user himself through a plurality of sensors, the agent in the smart glasses reasons out that the user is interested in a certain person through the whispering sound data "who is this person", the agent identifies the "certain person" in the surrounding environment within the range seen by the user through the image data, and based on this, the agent determines that the target object attracting the user's attention is the "certain person", and thus the agent uses the sensors to collect sensing data for the target object, such as focusing on collecting image data of the "certain person", thereby obtaining the second sensing data.

[0072] Step 104: executing at least one target instruction according to the first sensing data and the second sensing data.

[0073] In an implementation manner, the first sensing data and the second sensing data can be data reasoned by an agent deployed on the wearable device in step 104, such as behavior intention recognition of the wearable user, and then according to the recognized behavior intention of the wearable user, a corresponding target instruction is generated and executed, which can match the behavior intention of the wearable user.

[0074] For example, as shown in Figure 2As shown in the embodiment, the intelligent agent can recognize the behavior intention of the wearing user "want to know the basic information of a certain person", and thus the intelligent agent recognizes the "certain person" and obtains the "basic information of the certain person" through image recognition, data search and other processing procedures, so as to generate and execute the target instruction outputting the basic information such as the name, age and interpersonal relationship of the "certain person", and thus the "basic information of the certain person" can be provided to the wearing user, even if the wearing user temporarily forgets the identity of the "certain person", the wearing user can also learn it in time through the smart glasses, so as to avoid the situation of meeting strangers, thus meeting the current needs of the wearing user and improving the use experience of the smart glasses by the user.

[0075] For example, as shown in the embodiment, Figure 3 As shown in the embodiment, the intelligent agent recognizes the behavior intention of the wearing user "want to hear what the certain person says", and thus the intelligent agent recognizes the conversation content of the "certain person" through sound recognition, text generation and other processing procedures, so as to generate and execute the target instruction outputting the text "I heard A say, she heard B say..." of the conversation content, or generate and execute the sound "I heard A say, she heard B say..." of the conversation content, and thus the conversation content can be provided to the wearing user, even if the voice of the other party is small and the wearing user may not hear it clearly, the wearing user can also be conveyed through the smart glasses, so as to avoid the situation of wanting to hear but not being able to hear clearly, thus meeting the current needs of the wearing user and improving the use experience of the smart glasses by the user.

[0076] As can be seen from the above technical solutions, in the data processing method provided by the embodiment of the present application, after the first sensing data is collected through the sensor on the wearable device, the intelligent agent on the wearable device is used to perform data sensing on the first sensing data and attention recognition of the wearing user, and then the target object attracting the attention of the wearing user is determined. At this time, according to the target object, the second sensing data related to the target object is reacquired, and according to the first sensing data and the second sensing data of the target object attracting the attention of the wearing user, the target instruction for realizing the corresponding function is executed. As can be seen, in the present application, the target instruction is executed in combination with the sensing data of the object attracting the attention of the wearing user, which can make the realized function more matched with the matter concerned by the wearing user, so as to better meet the use needs of the user, and thus the wearable device can provide more accurate intelligent services for the user, and improve the use experience of the wearable device by the user.

[0077] In an implementation manner, when the second sensing data is obtained according to the target object in step 103, it can be realized by the following manner, as shown in the embodiment, Figure 4

[0078] Step 401: determining the object type to which the target object belongs.

[0079] ​The object type can include: a wearer himself / herself, a real environment in which the wearer is located (including a person or an object in the real environment), and a virtual environment viewed by the wearer (including a person or an object in the virtual environment).

[0080] Specifically, the data perception and object recognition of the agent can be used to determine the object type to which the target object belongs.

[0081] Step 402: obtaining a plurality of second sensing data in a target manner according to the target object.

[0082] The target manner corresponds to the object type. Different object types use different target manners to obtain the plurality of second sensing data.

[0083] In an implementation manner, the target manner can be a manner of extracting the second sensing data related to the target object from the first sensing data. Specifically, the agent extracts the sensing data related to the target object, i.e., the second sensing data, from the first sensing data. For example, the image data "features of a certain person" is extracted from the first sensing data, and for another example, the sound data "content of a certain person's speech" is extracted from the first sensing data.

[0084] In another implementation manner, the target manner can be a manner of controlling the sensor to re-collect. Specifically, the sensor is controlled to collect sensing data for the target object to obtain the plurality of second sensing data. For example, the camera is controlled to collect the image data of the features of "a certain person", and for another example, the microphone is controlled to collect the sound data of "a certain person's speech".

[0085] Specifically, in step 402, the sensor can be controlled to collect the sensing data related to the target object satisfying the association relationship with the first collection parameter, and the sensor can be controlled to collect the sensing data related to the target object satisfying the interference condition with the second collection parameter to obtain the target sensing data.

[0086] The data collection intensity of the sensor under the first collection parameter is higher than that under the second collection parameter.

[0087] For example, the details of the image data collected by the camera under 10 times of focal length are obviously more than those of the image data collected under 1 times of focal length. For another example, the sound volume of the sound data collected by the microphone under 10 times of gain is obviously greater than that of the sound data collected under 1 times of gain.

[0088] Sensor data that satisfies the association condition with the target object refers to sensor data belonging to the target object, such as sound signals emitted by the target object or image data of the target object. Sensor data that satisfies the interference condition with the target object refers to transmitted data that interferes with the sensor data belonging to the target object. For example, sound signals emitted by other people or objects in the surrounding environment interfere with the sound signals emitted by the target object; or the area occupied by other people or objects in the surrounding environment in the image captured by the camera is larger than the area occupied by the image of the target object.

[0089] As can be seen, in this embodiment, when acquiring the second sensory data, the first acquisition parameters can be used to enhance the acquisition of sensory data associated with the target object, while the second acquisition parameters can be used to weaken the acquisition of other sensory data that interferes with the target object. As a result, the resulting second sensory data can better represent the target object's state, such as the target object's words, facial expressions, etc. Based on this, when the intelligent agent executes the target instruction based on the first and second sensory data, it uses the second sensory data as the primary data and the first sensory data as a supplement to perform data reasoning to determine the wearer's behavioral intentions, and then generates and executes the corresponding target instruction to meet the wearer's current needs.

[0090] For example, Figure 5 As shown in the figure, 20 people are having dinner together at a large round table. User A is wearing smart glasses. B and C are located in front of user A on the left. B and C are chatting, but B and C are far away from A. The voices of other people around user A are also relatively loud. The smart glasses sense the movements of user A turning his head, tilting his ears, and lifting his hair to reveal his ears through the sensor data collected by the sensors. Based on this, it is recognized that user A's attention is on the chat content of B and C. At this time, the intelligent body in the smart glasses controls the microphone to collect the chat sound data of B and C with high gain, and collects the sound data of other people with low gain, so as to output "the chat content of B and C" through text or sound. For example, "the chat content of B and C" is projected in text on the screen of the smart glasses, or "the chat content of B and C" is output in sound through the smart headphones connected to the smart glasses to avoid information errors.

[0091] For example, Figure 6As shown in the hospital waiting room, the user wearing the smart glasses sits in the waiting room, and there are many patients waiting for the call number. The smart glasses perceive the action of the user wearing the smart glasses looking up to observe the call number screen through the sensor collected sensor data, and identify the attention of the user wearing the smart glasses on the name being called. At this time, the smart glasses control the microphone to collect the call number sound of the call station with high gain and collect the sound data of the surrounding patients with low gain, and then output the "call number content" through text or sound. Further, in the case of matching the name of the user wearing the smart glasses, the "call number content" is repeatedly output to avoid missing the number. For example, the "call number content" is projected on the screen of the smart glasses in a text manner, or the "call number content" is output in a sound manner through the smart earphone connected with the smart glasses.

[0092] For example, as shown in the figure, Figure 7 As shown in the figure, in the scene of riding at night, the user wearing the smart glasses rides a bicycle on the road. The visual environment is weak at night, and the smart glasses perceive the acceleration of the user wearing the smart glasses through the sensor collected sensor data, and identify that the attention of the user wearing the smart glasses is on the road ahead. At this time, the smart agent of the smart glasses controls the camera to collect image data in the road ahead of the user wearing the smart glasses and image data of the rear view angle with high pixels and large focal length, and collects image data of other view angles with low pixels and normal focal length. Thus, the smart agent performs data perception and data reasoning on these image data, and outputs "red light" and "rear two-wheeled vehicle approaching" through text or sound to prompt the user wearing the smart glasses to pay attention to safety.

[0093] Based on the above implementation scheme, in the case that the object type of the target object is the user wearing the smart glasses himself, the second sensor data collected in the embodiment can include the environment data in the blind area view angle of the user wearing the smart glasses.

[0094] Specifically, after determining that the target object is the user wearing the smart glasses, it can be known that the attention of the user wearing the smart glasses is focused on himself. At this time, the environment data in the blind area view angle of the user wearing the smart glasses can be collected in the embodiment, and the target instruction executed according to the environment data in the blind area view angle is used to output the first prompt information. The first prompt information is used to prompt the key object in the blind area view angle, such as other people approaching, vehicles approaching, approaching dangerous articles, red light state of the intersection, etc.

[0095] The blind area view angle of the user wearing the smart glasses can include the rear view angle of the user wearing the smart glasses in the case of good ambient light or the view angle blocked by obstacles. For example, as shown in the figure, Figure 8As shown in , the wearer is running on the side of the road, and the wearer's attention is focused on himself. At this time, the wearer cannot perceive the danger from the rear visually restricted area in the surrounding environment in time. Therefore, in this embodiment, the sensor on the wearable device can focus on collecting environmental data from the rear perspective of the wearer, and the intelligent agent can analyze the environmental data to determine whether there is any danger, thereby executing the target instruction to output text or sound "Please pay attention to the car coming from behind" or "Please pay attention to the stone behind", so as to remind the wearer to pay attention to his own safety.

[0096] Alternatively, the blind spot viewing angle of the user wearing the device may include: the viewing angle range at a specific distance from the user wearing the device when the ambient light is poor. Figure 9 As shown in , when the wearer rides at night, the night vision condition is weak, and the wearer listens to sports music through headphones. At this time, the wearer's attention is focused on himself, and the sports music in the headphones may interfere with the wearer's observation of the surrounding environment and hearing warning sounds such as horns. At this time, the wearer cannot accurately observe the status of traffic lights at distant intersections and the situation of other traffic participants. Therefore, in this embodiment, the sensor data on the wearable device is used to collect environmental data of distant intersections, and the environmental data is analyzed by the intelligent agent to determine the status of traffic lights and the situation of other traffic participants, so as to execute the target instruction to output text or sound of "Currently the light is red, no passage" and "Cars coming from the right", so as to remind the wearer to pay attention to his own safety.

[0097] In another implementation, when the target object is the wearer, the second sensory data collected in this embodiment may include vital sign data of the wearer. The target instruction corresponding to the vital sign data is used to output second prompt information, which is used to indicate the wearer's physical condition.

[0098] For example, Figure 10 As shown in the figure, the wearer brings smart headphones to participate in a marathon. During the race, the wearer focuses on running in the hope of improving his performance and may ignore his physical condition. Therefore, the intelligent agent in the smart headphones determines that the wearer's attention is on the running competition, and collects the wearer's vital signs data such as heart rate and blood pressure through sensors. Combined with the environmental data in the competition route, it outputs a voice message to the wearer, "The heart rate is normal, you can speed up in the current shade area, and there will be spray cooling after 500 meters to slow down and keep the heart rate below 160", thereby bringing the wearer a brand new marathon experience.

[0099] In an implementation, after the target object is determined, the target information associated with the target object is filtered from the knowledge base according to the user information of the wearing user, and the target information is outputted, in the case that the target object is a real environment in which the wearing user is located.

[0100] The user information of the wearing user represents the personal preference, basic information, and the like of the wearing user. The knowledge base can include a large amount of knowledge content, such as history, humanities, and the like.

[0101] For example, the wearing user carries the smart glasses to go shopping, and the smart agent on the wearable device determines that the wearing user focuses on a restaurant in the street. Then, the smart agent can filter the menu satisfying the user preference, such as sweet sweet and sour fish or spicy boiled fish, from the menu of the restaurant included in the knowledge base according to the eating preference of the wearing user, and output "there is boiled fish" through the loudspeaker in the smart glasses.

[0102] For another example, the wearing user carries the smart glasses to attend a meeting, and the smart agent on the wearable device determines that the wearing user focuses on the main speaker of the meeting. Then, the smart agent can filter the personal information of the main speaker of the meeting and the interpersonal network with the wearing user from the knowledge base according to the occupation and education background of the wearing user, and output "Yang **, whose published book *** you have purchased, and Yang ** and you both graduated from ** University" through the loudspeaker in the smart glasses.

[0103] In an implementation, after the target object is determined, the target information associated with the target object is filtered from the knowledge base according to the user information of the wearing user, and the target information is outputted, in the case that the target object is a real environment in which the wearing user is located.

[0104] The action prompt information is used to prompt the wearing user to perform an action on the virtual environment. The virtual environment can be a screen environment of an electronic device used by the wearing user, such as a game interface or a ticket booking interface in a mobile phone screen, a conference interface in a notebook screen, and the like.

[0105] For example, taking the virtual environment as the interface of a mobile phone game, the wearing user carries the smart glasses to play the mobile phone game, the agent in the smart glasses determines that the wearing user focuses on the mobile phone game, and according to the game interface output by the mobile phone game and the input operation of the wearing user on the game interface, the operation intention of the wearing user on the mobile phone game can be determined, such as wanting to realize a certain attack, based on which, the agent can output the sound "click the left upper corner symbol @, and then click the right upper corner symbol $, to unlock the attack X" through the smart earphone connected to the smart glasses, or output the text "click the left upper corner symbol @, and then click the right upper corner symbol $, to unlock the attack X" through the lens screen of the smart glasses, thereby prompting the wearing user to perform the corresponding action on the mobile phone game.

[0106] In an implementation manner, the agent can also process the first sensing data in the embodiment, such as environment type identification, to determine the environment type of the environment where the wearable device is located, such as a road driving environment, a construction environment, a scenic spot browsing environment, etc., and output the risk prompt information in the case that the environment type matches the preset alarm type.

[0107] For example, in the case that the environment type is a construction environment, the agent outputs "construction area, pay attention to safety" through the loudspeaker connected or contained by the wearable device; for example, in the case that the environment type is a scenic spot browsing environment, the agent outputs "scenic spot is crowded, prevent trampling" through the screen connected or contained by the wearable device.

[0108] In an implementation manner, after determining the target object, the target object and the first sensing data and the second sensing data can be stored according to the relationship between the target object and the first sensing data and the second sensing data. That is, the target object focused on by the wearing user and the corresponding sensing data are associated and recorded in the embodiment.

[0109] For example, the wearing user carries the smart glasses to browse the scenic spot, the agent in the smart glasses associates and records each scenic spot landscape focused on by the wearing user with the corresponding sensing data (landscape image, tour guide explanation, etc.) in a time sequence or according to the position relationship between the scenic spot landscapes, to form a browsing log of the wearing user in the scenic spot.

[0110] Reference Figure 11 A structure schematic diagram of a data processing apparatus provided in the embodiment of the present application can be deployed in a wearable device, such as a smart earphone, smart glasses, etc. The data processing apparatus can include the following units:

[0111] The first obtaining unit 1101 is configured to obtain a plurality of first sensing data through a plurality of sensors deployed on the wearable device;

[0112] The object determining unit 1102 is configured to determine a target object by using an agent deployed on the wearable device to perform data sensing on the first sensing data and to identify the attention of the wearer.

[0113] The second obtaining unit 1103 is configured to obtain a plurality of second sensing data according to the target object; the second sensing data is related to the target object.

[0114] The instruction executing unit 1104 is configured to execute at least one target instruction according to the first sensing data and the second sensing data.

[0115] As can be seen from the above technical solution, in the data processing apparatus provided by the embodiment of the present application, after the first sensing data is collected by the sensor on the wearable device, the agent on the wearable device is used to perform data sensing on the first sensing data and to identify the attention of the wearer, and then the target object attracting the attention of the wearer is determined. At this time, according to the target object, the second sensing data related to the target object is re-obtained. According to the first sensing data and the second sensing data related to the target object attracting the attention of the wearer, the target instruction for realizing the corresponding function is executed. It can be seen that, in the present application, the target instruction is executed in combination with the sensing data of the object attracting the attention of the wearer, which can make the realized function more match the matter attracting the attention of the wearer, thereby more satisfying the use demand of the wearer, and further making the wearable device provide more accurate intelligent service for the wearer and improving the use experience of the wearer on the wearable device.

[0116] In an implementation manner, the second obtaining unit 1103 is specifically configured to determine an object type to which the target object belongs; the object type includes the wearer himself / herself, a real environment in which the wearer is located, and a virtual environment watched by the wearer; and a plurality of second sensing data is obtained in a target manner according to the object type; the target manner corresponds to the object type.

[0117] The target manner includes one of the following: controlling the sensor to collect the sensing data for the target object to obtain a plurality of second sensing data; and extracting the second sensing data related to the target object from the first sensing data.

[0118] In an implementation manner, the second obtaining unit 1103 is specifically configured to: control the sensor to collect, in a first collection parameter, the sensing data associated with the target object, and control the sensor to collect, in a second collection parameter, the sensing data satisfying the interference condition with the target object, to obtain the second sensing data; and the data collection intensity of the sensor in the first collection parameter is higher than that in the second collection parameter.

[0119] In a case where the object type is the wearer himself / herself, the second sensing data comprises one of:

[0120] environmental data in a blind area of the wearer, the target instruction corresponding to the environmental data being configured to output first prompt information, the first prompt information being configured to prompt a key object in the blind area of the wearer;

[0121] vital sign data of the wearer, the target instruction corresponding to the vital sign data being configured to output second prompt information, the second prompt information being configured to prompt a physical state of the wearer.

[0122] In an implementation manner, the apparatus in the embodiment can further comprise the following units, as shown in Figure 12

[0123] The information output unit 1105 is configured to: after the object determining unit 1101 determines the target object by using the agent deployed on the wearable device to perform data sensing on the first sensing data and attention recognition of the wearer, in a case where the target object is a real environment in which the wearer is located, filter target information associated with the target object from a knowledge base according to user information of the wearer; and output the target information.

[0124] In an implementation manner, the information output unit 1105 is further configured to: after the object determining unit 1101 determines the target object by using the agent deployed on the wearable device to perform data sensing on the first sensing data and attention recognition of the wearer, in a case where the target object is a virtual environment viewed by the wearer, determine an operation intention of the wearer to the virtual environment according to content output by the virtual environment and an input operation of the wearer to the virtual environment; output action prompt information according to the operation intention; and the action prompt information is configured to prompt the wearer to perform an action on the virtual environment.

[0125] ​In an implementation manner, the information output unit 1105 is further configured to: process the first sensing data by using the agent to determine an environment type of an environment where the wearable device is located; and output a risk prompt information in a case where the environment type matches a preset alarm type.

[0126] It should be noted that the specific implementation manners of the units in this embodiment can refer to the corresponding contents in the foregoing, which will not be described in detail here.

[0127] Reference Figure 13 A structural schematic diagram of a wearable device is provided in the embodiment of the present application, which can be a smart earphone, smart glasses, etc. The wearable device can include the following structures:

[0128] A plurality of sensors 1301 are configured to collect a plurality of first sensing data.

[0129] A processor 1302 is configured to deploy an agent, and the agent is configured to perform data sensing on the first sensing data by using the agent deployed on the wearable device and to identify attention of a wearer to determine a target object; to obtain a plurality of second sensing data according to the target object; the second sensing data is related to the target object; and to execute at least one target instruction according to the first sensing data and the second sensing data.

[0130] Specifically, the wearable device can include or be connected to a display screen, and output text indicated by the target instruction, such as the first prompt information and the second prompt information, through the display screen. The wearable device can include or be connected to a loudspeaker, and output sound output by the target instruction, such as the first prompt information and the second prompt information, through the loudspeaker.

[0131] The sensor 1301 includes at least two of the following:

[0132] An image sensor, a sound sensor, a motion sensor, a biological sensor, and an environment sensor.

[0133] The image sensor is configured to collect image data.

[0134] The sound sensor is configured to collect sound data.

[0135] The motion sensor is configured to collect motion data of the wearable device and / or body posture data of the wearer.

[0136] The biological sensor is configured to collect vital sign data of the wearer.

[0137] The environment sensor is configured to collect environment data of an environment where the wearable device is located.

[0138] As can be seen from the above technical solution, in a wearable device provided by an embodiment of the present application, after first sensor data is collected by a sensor on the wearable device, an intelligent agent on the wearable device performs data perception on the first sensor data and identifies the wearer's attention, thereby determining a target object that attracts the wearer's attention. At this time, based on the target object, second sensor data related to the target object is re-acquired. Based on this target object, a target instruction for implementing a corresponding function is executed based on the first sensor data and the second sensor data related to the target object that attracts the wearer's attention. It can be seen that in this application, executing the target instruction in conjunction with the sensor data of the object that attracts the wearer's attention can make the implemented function more consistent with the things that the wearer is concerned about, thereby better meeting the user's usage needs, thereby enabling the wearable device to provide the user with more accurate intelligent services and improve the user's experience of the wearable device.

[0139] In order to enrich the intelligent experience that wearable devices bring to wearable users, this application provides a wearable device equipped with a personal intelligent body. The wearable device (also called a wearable device) is a new personal artificial intelligence (AI) companion wearable product with the core experience of "photographic memory, responsiveness at any time, and in-depth companionship". It can actively perceive user needs and provide active, personalized, real-time, and reliable intelligent support in multiple scenarios such as life, study, work, and social interaction, serving as an all-weather AI portable assistant.

[0140] The wearable device has the ability to carry personalized data, can obtain sound, images and videos, provide auditory, visual, tactile perception and interaction capabilities, and can achieve multimodal intelligent perception and auxiliary decision-making of the user's environment and behavior. Figure 14 As shown in the , taking smart headphones as an example, the device integrates a camera module, a microphone array, a speaker, a display (a projection screen implemented by the headphones or a display on the headphone box), a pressure-sensitive touch module, a head motion recognition device, a physiological data monitoring sensor and a multimodal AI processing unit to build an intelligent perception system that integrates environmental perception, user status analysis and natural interaction.

[0141] In a specific solution, this application enhances the wearer's visual and auditory abilities through sensor layout:

[0142] a) Identify user’s weak language input: Accurately perceive and understand the user’s whispered expression, and understand the user’s potential or implicit needs.

[0143] b) Natural language and action interaction with the device: including natural gestures (such as hair-raising action) to trigger specific functions, and head direction change (such as tilting the ear, turning the head) to trigger directional hearing enhancement, focusing on sound collection, and building a commandless human-computer interaction mechanism.

[0144] c) Visual semantic understanding of the environment and auxiliary perception: through visual-voice cooperative understanding of environmental semantics, such as "user says", "what is on the left", the agent gives accurate feedback through visual processing and spatial mapping, and improves the user's spatial perception ability.

[0145] Moreover, the application enhances the reminder through multi-modal fusion and dynamic perception blind area:

[0146] a) When the user's visual field is limited behind or at night, the system can start hearing enhancement and visual enhancement, and automatically start environmental danger monitoring.

[0147] b) The core focus area of hearing is 0, -45 in front, and the non-core perception collection area such as warning sound (such as whistle, calling name) can be enhanced according to the relevance to the user. Hearing and vibration feedback.

[0148] c) Combined with visual and auditory information, timely feedback enhancement, in unfamiliar and unfamiliar places, automatically strengthen the information acquisition of the unfamiliar environment to the user.

[0149] It can be seen that the application constructs natural and invisible human-computer interaction through multi-modal data perception (vision, hearing, display, head movement, interaction), and the application supports the perception understanding of micro-language input to realize the intelligent response of invisible demand; in addition, the application improves the convenience of wearing interaction through the non-sensing trigger based on action and posture; at the same time, the application improves the completeness of the user's information perception through the dynamic blind area completion and enhancement of multi-modal data fusion; finally, the application provides high-frequency, structured, multi-source environmental data for the agent, and strengthens its understanding and reaction ability to user's intention and scene.

[0150] As shown in Figure 15 , it is an implementation framework diagram of a wearable device, wherein:

[0151] In the power management module, a lithium polymer battery LI-PO (Lithium Polymer Battery), wireless charging, a power management integrated circuit PMIC (Power Management integrated circuit) are deployed; in the connection transmission module, WiFi, Bluetooth BT (blue tooth), a global positioning system GPS (Global Positioning System), a fifth generation mobile communication technology 5G (5th Generation Mobile Communication Technology) are deployed; in the storage module, a low-power double data rate synchronous dynamic random access memory LP DDR (‌Low Power Double Data Rate‌), an embedded multi-media card Emmc (Embedded Multi Media Card), a universal flash storage UFS (Universal Flash Storage) are deployed; in the calculation module, a combination chip, a micro control unit MCU (Microcontroller Unit), a neural network processing unit NPU (Neural network Processing Unit), an image signal processor ISP (mage Signal Processing), a secure enclave are deployed; in the data acquisition and perception module, a camera, a millimeter wave radar, an IMU, an environmental sensor, a bio sensor are deployed; in the interaction module, a microphone, a loudspeaker (sound or bone conduction), a display screen, a tactile feedback sensor, an indicator light are deployed, and an operating system is arranged above the interaction module, for receiving user interaction data, cloud interaction, intent recognition, etc.

[0152] The data processing flow of the wearable device is as shown in Figure 16

[0153] Through semantic perception such as voice, gesture, touch recognition arousal, the intelligent agent acquires (Sense) and cooperates with the cloud large model to recognize (Know) the scene, and then performs automatic task planning (Act), that is, executes the target instruction. Further, feedback to the semantic perception to synchronize the memory and knowledge base.

[0154] The intelligent agent performs automatic task planning, queries, records, forms a to-do list (Todolist) content, and outputs to the wearable user, broadcasts or presents to the display screen, and performs cross-device memory and knowledge base synchronization.

[0155] Taking the wearable device as an example, the technical solutions of the present application are described as follows:​

[0156] For example, as shown in Figure 17 , the earphone and camera are integrated, and the wearable device is worn on the head as a whole. The smart earphone includes a support, a left earphone, and a right earphone. The left earphone is provided with a microphone, a loudspeaker, a bio sensor, a touch feedback sensor, and a camera. The right earphone is provided with a camera. The support is provided with a battery. The smart body can be arranged at any position in the support, the left earphone, and the right earphone.

[0157] For another example, as shown in Figure 18 , the earphone and camera are separated. The camera can be arranged on the earphone box or the earphone. The camera, the microphone, the display screen, the loudspeaker, and the sensor are arranged on the earphone box. The earphone box undertakes the interaction of data synchronization and display.

[0158] Based on the implementation framework of the wearable device, as shown in Figure 19 , the architecture composition and the processing flow, the technical scheme of the present application can be implemented through the following modules:

[0159] I. Data acquisition module, mainly used for multi-modal sensing data acquisition of the wearable device:

[0160] The following original sensing data (i.e., first sensing data) is collected through the sensors integrated in the wearable device:

[0161] (1) Visual data, i.e., image data (such as a camera);

[0162] (2) Sound data (such as a microphone);

[0163] (3) User state data (such as head movement, heart rate, and other physiological parameters);

[0164] (4) User interaction behavior data (such as head deflection and hand gestures);

[0165] II. Data processing module

[0166] 1. Environmental perception and safety information extraction:

[0167] (1) Voice feedback data (such as keyword triggering and tone change);

[0168] (2) Based on visual and sound data, identify the environment in which the user is located (whether it is strange, noisy, risky, etc.).

[0169] (3) Risk warning preparation for specific scenes (such as roads and construction zones).

[0170] 2. User attention and interaction state recognition:

[0171] (1) Real-time analysis of user head movement behavior, whether frequent head shaking, daydreaming, etc.

[0172] (2) If combined with EEG, the user's focus / looseness state can also be extracted synchronously.

[0173] Three, perceptual enhancement and attention assistance mechanism

[0174] First, the user's attention is the user's own data collection ability and understanding ability.

[0175] 1. Perception blind area enhancement data information acquisition (i.e. second sensor data), with attention focus on the user himself as a condition: for example: movement, appearance, user's own state, feeling heartbeat, etc.

[0176] (1) Detect that the user has ignored important environmental information (such as safety signs), and the agent automatically triggers the information enhancement mechanism to prompt the user to pay attention, which can be voice or visual. For example, when the user is in a construction site, prompt the user of the danger.

[0177] (2) Based on "user unattended area" to prioritize screening intervention content and reduce information interference. For example, the sound of the user's attention to the speaker is enhanced and collected, and the surrounding noise is shielded, and the interference information is weakened.

[0178] (3) The user pays attention to his own state as a label record.

[0179] 2. User attention area understanding and personalized content enhancement mechanism, with attention switching as the environment for adjustment:

[0180] (1) Judgment: such as user side head listening attention, active interaction (brushing hair, buttons such as user clicking on the button on the earphone, etc.);

[0181] (2) If the user continuously pays attention to an object or scene, the system automatically matches personalized supplementary information (such as the history of a certain place, culture, etc.) based on the knowledge graph / historical preferences, which can be used in cultural tourism, learning companionship, conference notes, etc. Field.

[0182] (3) If the user's attention is a virtual environment, combined with the device's data input, it helps to understand the user's intention, such as voice and application interface operation coordination, chatting, buying tickets, booking seats, etc.

[0183] (4) The user's current attention content and historical interaction content are associated and recorded based on semantic association, realizing the "memory synchronization" function based on attention changes, providing support for subsequent summary, review, and judgment.

[0184] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0185] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0186] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0187] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, comprising: Obtaining multiple first sensor data through multiple sensors deployed on the wearable device; Utilizing an agent deployed on the wearable device to perform data perception on the first sensor data and identify the attention of the wearer to determine a target object; According to the target object, a plurality of second sensing data are obtained; the second sensing data are related to the target object; At least one target instruction is executed according to the first sensing data and the second sensing data.

2. The method according to claim 1, wherein obtaining a plurality of second sensor data according to the target object comprises: Determine the object type to which the target object belongs; The object types include: the wearing user himself, the real environment where the wearing user is located, and the virtual environment viewed by the wearing user; acquiring a plurality of second sensor data in a targeted manner according to the object type; The target mode corresponds to the object type.

3. The method according to claim 2, wherein the target mode comprises one of the following: Controlling the sensor to collect sensing data from the target object to obtain a plurality of second sensing data; Second sensory data related to the target object is extracted from the first sensory data.

4. The method according to claim 3, wherein the step of controlling the sensor to collect sensing data from the target object to obtain a plurality of second sensing data comprises: Controlling the sensor to collect sensor data that satisfies an association relationship with the target object using a first acquisition parameter, and controlling the sensor to collect sensor data that satisfies an interference condition with the target object using a second acquisition parameter, so as to obtain a plurality of second sensor data; The data acquisition intensity of the sensor under the first acquisition parameter is higher than the data acquisition intensity under the second acquisition parameter.

5. The method according to claim 2, wherein when the object type is the wearing user, the second sensory data comprises one of the following: Environmental data within the blind spot of the user wearing the wearer, the target instruction corresponding to the environmental data is used to output first prompt information, and the first prompt information is used to prompt a key object within the blind spot; The vital sign data of the wearing user, the target instruction corresponding to the vital sign data is used to output second prompt information, and the second prompt information is used to prompt the physical condition of the wearing user.

6. The method according to claim 1, after using an agent deployed on the wearable device to perform data perception on the first sensor data and identify the wearer's attention to determine a target object, the method further comprises: In a case where the target object is a real environment where the wearing user is located, filtering target information associated with the target object in a knowledge base according to the user information of the wearing user; The target information is output.

7. The method according to claim 1, after utilizing an agent deployed on the wearable device to perform data perception on the first sensor data and identify the wearer's attention to determine a target object, the method further comprises: When the target object is a virtual environment viewed by the wearing user, determining the wearing user's operation intention on the virtual environment based on content output by the virtual environment and input operations of the wearing user on the virtual environment; According to the operation intention, action prompt information is output; the action prompt information is used to prompt the wearing user to perform an action on the virtual environment.

8. The method according to claim 1, further comprising: Processing the first sensor data using the agent to determine an environment type of the wearable device; When the environment type matches the preset alarm type, risk warning information is output.

9. A wearable device comprising: Multiple sensors, used for collecting multiple pieces of first sensor data; a processor, configured to deploy an intelligent agent, wherein the intelligent agent is configured to perform data perception on the first sensor data and attention recognition of the wearer using the intelligent agent deployed on the wearable device to determine a target object; According to the target object, a plurality of second sensing data are obtained; the second sensing data are related to the target object; and according to the first sensing data and the second sensing data, at least one target instruction is executed.

10. The wearable device according to claim 9, wherein the sensor comprises at least two of the following: Image sensors, sound sensors, motion sensors, biosensors, environmental sensors; in, The image sensor is used to collect image data; The sound sensor is used to collect sound data; The motion sensor is used to collect motion data of the wearable device and / or body posture data of the wearable user; The biosensor is used to collect vital sign data of the wearer; The environmental sensor is used to collect environmental data of the environment in which the wearable device is located.

Images