A task execution method and electronic device
By evaluating the applicability of perception data associated with target users and filtering noisy data, the problem of inaccurate task execution by electronic devices in multimodal perception data is solved, achieving higher task accuracy and resource efficiency.
Patent Information
- Application Number
- CN202611339625.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-31
- Publication Date
- 2026-11-17
AI Technical Summary
In the prior art, electronic devices are prone to introducing noisy data when performing tasks based on multi-modal sensing data, which can lead to inaccurate or erroneous task execution.
By identifying the target feature information of the perceived data associated with the target user, assessing its applicability, and performing tasks using only the perceived data that meets the conditions, noisy data is filtered out.
It improves the accuracy and reliability of task processing, reduces resource consumption, and avoids unnecessary data processing and collection.
Smart Images

Figure CN123114592A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a task execution method and an electronic device. Background Technology
[0002] Electronic devices can perform various tasks with the help of intelligent programs or artificial intelligence models, such as intelligent agents. To ensure reliable task execution, electronic devices need to process data based on different modalities of the user, such as audio, video, and physiological signals. However, uniformly processing sensory data collected from various modal input channels can easily introduce noise data irrelevant to the task, often resulting in inaccurate or erroneous task execution. Summary of the Invention
[0003] On the one hand, this application provides a task execution method, including:
[0004] The target task to be performed and the first perception data of at least one modality required to perform the target task are determined, wherein the first perception data is perception data associated with the target user;
[0005] Determine the target feature information of the first perceived data in its modality, wherein the target feature information is used to characterize at least one of the semantic features and user features expressed by the first perceived data;
[0006] Based on the target feature information corresponding to the first sensing data, the applicability of the first sensing data to the target task is determined;
[0007] The target task is performed using at least a portion of the first perception data in which the applicability satisfies the target conditions.
[0008] In one possible implementation, the target feature information includes: user feature information;
[0009] The step of determining the applicability of the first sensing data to the target task based on the target feature information corresponding to the first sensing data includes:
[0010] Based on the user characteristic information of the first sensing data, determine the possibility that the first sensing data belongs to the valid sensing data that the target user expects to collect.
[0011] Based on the probability corresponding to the first sensing data, the applicability of the first sensing data to the target task is determined.
[0012] In yet another possible implementation, the target feature information includes: semantic feature information;
[0013] The step of determining the applicability of the first sensing data to the target task based on the target feature information corresponding to the first sensing data includes:
[0014] Determine the degree of matching between the semantic feature information of the first sensing data and the semantic feature information of other first sensing data;
[0015] Based on the matching degree corresponding to the first sensing data, the applicability of the first sensing data to the target task is determined.
[0016] In another possible implementation, the task execution method also includes:
[0017] Determine the task characteristic information of the target task;
[0018] Based on the task feature information of the target task and each of the first perception data, the correlation between the first perception data and the target task is determined;
[0019] Based on the correlation of the first sensing data, the applicability of the first sensing data is corrected.
[0020] In another possible implementation, determining the correlation between the first sensing data and the target task based on the task feature information of the target task and each of the first sensing data includes:
[0021] Based on the first perception data, the scene characteristics of the target user are determined;
[0022] Based on the task feature information of the target task and the scene features, the correlation between the first perception data and the target task is determined.
[0023] In another possible implementation, determining the matching degree between the semantic feature information of the first perceived data and the semantic feature information of other first perceived data includes:
[0024] Based on the semantic feature information of the first perceptual data and at least one other modality of first perceptual data, common information of items jointly expressed by multiple modalities is determined;
[0025] Based on the semantic feature information of the first perceived data, the matching degree between the item information expressed by the first perceived data and the common item information is determined.
[0026] In yet another possible implementation, the first sensing data of the at least one modality includes at least one of the following:
[0027] At least one audio data stream;
[0028] At least one video feed;
[0029] At least one physiological signal of the target user;
[0030] Determining the user characteristic information of the first perceived data includes at least one of the following:
[0031] In response to the first perceived data including audio data, voiceprint features in the audio data are determined;
[0032] In response to the first perceived data including video data, the viewpoint orientation features of the target user represented by the video data are determined;
[0033] In response to the first sensed data including the physiological signal, at least one of the variation characteristics and peak values of the physiological signal is determined.
[0034] In another possible implementation, based on the user characteristic information of the first perceived data, determining the probability that the first perceived data belongs to the valid perceived data that the target user intends to collect includes at least one of the following:
[0035] In response to the first perceived data being audio data, based on the matching degree between the voiceprint features in the audio data and the configured user voiceprint features, the possibility that the audio data is audio data emitted by the target user is determined;
[0036] In response to the first perceived data being video data, based on the motion vector represented by the motion data and the viewpoint direction features, it is determined that the video data may include content that the target user is actively paying attention to; wherein, the motion data belongs to the second perceived data required to perform the target task, and the motion data is used to represent the motion state corresponding to the target electronic device used by the target user;
[0037] In response to the first perceived data being the physiological signal, based on at least one of the variation characteristics and peak values of the physiological signal, the probability that the physiological signal is a physiological signal generated under external stimulation is determined.
[0038] In another possible implementation, performing the target task using at least a portion of the first perceptual data in which the applicability satisfies the target condition includes:
[0039] If the applicability of the first sensing data is not lower than the first threshold, the first sensing data is determined as the target sensing data required to process the target task;
[0040] If the applicability of the first sensing data is not lower than the second threshold and is lower than the first threshold, the first sensing data is compressed, and the first sensing data obtained by the compression is determined as the target sensing data required to process the target task, wherein the first threshold is greater than the second threshold.
[0041] Based on the determined target perception data, the target task is executed.
[0042] In another aspect, this application also provides an electronic device, including: a processor and a memory;
[0043] The processor is configured to: determine a target task to be executed and first perceptual data of at least one modality required to execute the target task, wherein the first perceptual data is perceptual data associated with a target user; determine target feature information of the first perceptual data in its modality, wherein the target feature information is used to characterize at least one of semantic features and user features expressed by the first perceptual data; determine the applicability of the first perceptual data to the target task based on the target feature information corresponding to the first perceptual data; and execute the target task using at least a portion of the first perceptual data whose applicability satisfies the target conditions.
[0044] The memory is used to store the programs required by the processor to perform operations. Attached Figure Description
[0045] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.
[0046] Figure 1 A flowchart illustrating the task execution method provided in this application;
[0047] Figure 2 Another flowchart illustrating the task execution method provided in this application;
[0048] Figure 3 Another flowchart illustrating the task execution method provided in this application;
[0049] Figure 4 Another flowchart illustrating the task execution method provided in this application;
[0050] Figure 5 Another flowchart illustrating the task execution method provided in this application;
[0051] Figure 6An example diagram illustrating the implementation process of the task execution method provided in this application;
[0052] Figure 7 A schematic diagram of the component architecture of the electronic device provided in this application. Detailed Implementation
[0053] The solution proposed in this application is applicable to scenarios in which task processing is performed based on at least one modality of sensory data associated with the user. For example, it can combine sensory data of different modalities, such as the user's heart rate and other physiological signals, video data collected from the user's perspective, and audio data of the user's environment. It can also combine schedules or files recently operated by the user to perform tasks such as travel reminders, office tasks, generating work plans, or mood fluctuation reminders.
[0054] The solution proposed in this application can improve the accuracy and reliability of task processing.
[0055] The solution in this application can be applied to electronic devices, including but not limited to laptops, mobile phones, smart glasses, or smartwatches, without any specific restrictions.
[0056] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.
[0057] As will be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0058] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.
[0059] like Figure 1 This is a flowchart illustrating the task execution method provided in this application. The method in this embodiment may include:
[0060] S101, determine the target task to be performed and the first perception data of at least one modality required to perform the target task.
[0061] The target task can be a task actively generated by the electronic device. For example, the electronic device determines the task to be executed based on at least a portion of the following: the user's historical habit data, the user's recent operational behavior data (such as opened files, notes, and application operation data), historical sensory data associated with the user, the first sensory data, and other sensory data collected by the electronic device or other electronic devices used by the user. This application does not impose any restrictions on the specific implementation of the target task determined by the electronic device.
[0062] The target task can also be a task to be executed that is input by the user, such as a target task indicated by the user through instructions.
[0063] The target task can be of various types, such as a trip reminder task to remind you of your travel itinerary, an information retrieval task to retrieve information based on input information, an office task, or a question-and-answer task, etc. There are no specific restrictions.
[0064] The first perception data is perception data that is associated with the target user.
[0065] For example, the first perception data can be perception data of the target user's field of vision or environment collected by sensors, such as images or videos of the user's field of vision collected by image sensors.
[0066] For example, the first perception data can be perception data of the target user's environment collected by sensors. For instance, audio data of the target user's environment collected by an audio sensor may include the voice spoken by the target user and noise data.
[0067] For example, the first sensing data can be sensing data obtained by using sensors with the target user as the sensing object. For instance, the first sensing data can be the target user's heart rate, skin conductance, or other physiological signals collected by a heart rate sensor, skin conductance sensor, or other physiological signal collector. Skin conductance (also known as skin conductance signal) is used to characterize the skin's ability to conduct weak electrical currents. Another example is the target user's movement speed sensed by a speed sensor in a wearable electronic device.
[0068] Of course, there are other possibilities for the first perception data associated with the target user, and there are no restrictions on this.
[0069] The target user is the user served by the target task. This could be a user pre-bound to an electronic device, or the user of the electronic device itself. For example, if the target task is a trip reminder task, then the trip reminder task is either generated for the target user or generated by input or triggered by the target user.
[0070] The modality of the first sensing data is used to characterize the data type and source. For example, the first sensing data can be video data collected by an image sensor, audio data collected by an audio sensor, or physiological signals collected by a physiological sensor.
[0071] In this application, the first sensing data of at least one modality can be sensing data obtained by sensors on the electronic device; it can also be sensing data sensed and transmitted to the electronic device by at least one other electronic device. In practical applications, some of the first sensing data of at least one modality may originate from the sensors of the electronic device, while other portions may originate from at least one other electronic device. This application does not limit the specific source of the first sensing data.
[0072] For example, in one instance, the electronic device only performs task execution and is not responsible for sensing data related to the target user. In this case, the electronic device can obtain the first sensing data from at least one of the electronic devices worn by the target user, such as smart glasses, smartwatches, and mobile phones used by the target user.
[0073] In another example, the electronic device can not only perform tasks but also collect initial sensor data and receive initial sensor data sent by other electronic devices. For instance, if the electronic device is a mobile phone, then during the target user's use of the phone, the phone can collect video images and audio data, including the target user's voice, from the target user's perspective. Simultaneously, the phone can also receive physiological signals, video images, or motion data sent by the target user's smart glasses or smartwatch.
[0074] It should be noted that in this application, there may be one or more data sources for each modality. Therefore, the first sensing data for each modality may include one or more, but the processing procedure for any first sensing data in a modality is the same.
[0075] S102, for each first sensing data, determine the target feature information of the first sensing data in its respective modality.
[0076] The target feature information is used to characterize at least one of the semantic features and user features expressed by the first perceived data.
[0077] The semantic features expressed by the first perception data are used to represent the deeper semantic content revealed by the first perception data. For example, semantic features can represent the scene state and event information reflected by the first perception data.
[0078] The user features expressed by the first perception data are those features in the first perception data that can characterize the identity, state, or behavior of the target user. For example, the user features expressed by the first perception data may include, but are not limited to, the user's fingerprints, physiological state, and current perspective.
[0079] When the modality of the first sensing data is different, the type and specific content of the target feature information determined from the first sensing data may be different.
[0080] S103, for each first perception data, based on the target feature information corresponding to the first perception data, determine the applicability of the first perception data to the target task.
[0081] The applicability score characterizes the degree to which the first sensing data is suitable for processing the target task. The applicability score reflects the importance of the first sensing data in performing the target task.
[0082] S104, using at least a portion of the first perception data whose applicability satisfies the target conditions, perform the target task.
[0083] Among them, the applicability meeting the target condition indicates that the first perception data is the effective data required to process the target task.
[0084] The target condition can be set according to actual needs. For example, the applicability can be satisfied by the applicability being greater than the set target threshold, or by the applicability falling within the set threshold range.
[0085] In this application, after determining at least a portion of the first perceptual data used to perform the target task, performing the target task based on the determined perceptual data can be done using an artificial intelligence model or an intelligent agent. No restrictions are placed on the specific implementation method for performing the target task.
[0086] Of course, when performing the target task, in addition to using the first perception data, it may also be necessary to use other perception data that is unrelated to the target user or the user's operation data on the electronic device or other electronic devices. This operation data may include files associated with the user, such as files operated by the user or files generated in the user's history, etc., without any specific restrictions.
[0087] In this embodiment, after obtaining the first perception data associated with the target user, the applicability of the first perception data to the target task is determined based on the target feature information of the first perception data in its modality. The target task is then executed using at least a portion of the first perception data whose applicability meets the target conditions. This effectively filters out invalid perception data that is not applicable to the target task, reduces the noise data contained in the various perception data used to execute the target task, and thus reduces the possibility of inaccurate or erroneous task execution, improving the accuracy and reliability of task execution.
[0088] Furthermore, in the process of performing the target task in this application, only the perception data applicable to the target task is processed, thereby avoiding in-depth processing of perception data whose applicability does not meet the target conditions, and reducing the amount of resource processing required to perform the target task.
[0089] In addition, since this application only needs to use the first sensing data that meets the target conditions for suitability to perform the target task, the data acquisition of the relevant sensors can be terminated for the first sensing data that does not meet the target conditions for suitability, so as to avoid the continuous operation and collection of invalid sensing data, which would lead to the consumption of power and other resources.
[0090] In this application, for the first perception data whose applicability meets the target conditions, all of the first perception data can be used as perception data for performing the target task, or, depending on the applicability, some or all of the data can be determined from the first perception data to perform the target task.
[0091] For example, in one possible implementation, if the applicability of the first sensing data is not lower than a first threshold, the first sensing data is determined as the target sensing data required to process the target task; if the applicability of the first sensing data is not lower than a second threshold but lower than the first threshold, the first sensing data is compressed, and the compressed first sensing data is determined as the target sensing data required to process the target task. Here, the first threshold is greater than the second threshold. Accordingly, the target task can be executed based on the determined target sensing data.
[0092] There are various ways to compress the first sensing data, such as downsampling, extracting summary information, or extracting key data. The specific compression method can be different depending on the modality of the first sensing data.
[0093] For example, if the first sensing data is video or audio data, it can be downsampled to reduce the number of video or audio frames, and the downsampled first sensing data can be used as the target sensing data. If the first sensing data is audio data, it can also be a summary of the text converted from the audio data, and the summary information can be used as the target sensing data to reduce the amount of audio data. If the first sensing data is a physiological signal, at least one of the highest value, lowest value, and trend information of the physiological signal can be determined as the target sensing data.
[0094] It is understandable that if the applicability of the first perception data is lower than the second threshold, it means that the applicability of the first perception data does not meet the target conditions. In this case, it can be determined that the first perception data is not the perception data required to perform the task, and the first perception data is masked as noise data.
[0095] Of course, in order for users to correct the first perception data that is misjudged as noise data and to reasonably adjust the first threshold and the second threshold, if the applicability of the first perception data does not meet the target conditions, this application may also store the metadata information of the first perception data. The metadata information may include, but is not limited to, the timestamp, modality type and the reason why the first perception data is identified as noise data.
[0096] In one alternative approach, considering that the sensitivity of the effectiveness of the perceived data varies depending on the type of the scenario, this application may first determine the scenario type of the target user based on each first perception data, and then determine the first threshold and the second threshold corresponding to the scenario type.
[0097] In this application, the target feature information of the first perceived data can be one or more of the semantic features and user features expressed by the first perceived data. The solution of this application will be introduced below in combination with different situations of target feature information.
[0098] The following example uses target feature information, including user feature information, in conjunction with... Figure 2 Please provide an explanation. For example... Figure 2 This is a flowchart illustrating the task execution method provided in this application. The method in this embodiment may include:
[0099] S201, determine the target task to be performed and the first perception data of at least one modality required to perform the target task.
[0100] The first perception data is perception data that is associated with the target user.
[0101] S202, for each first perception data, determine the user feature information of the first perception data in its respective modality.
[0102] Among them, user feature information is feature information determined based on the first perception data, used to characterize the identity, status or behavior of the target user.
[0103] When the modality of the first sensing data is different, the specific feature type of the user feature information can also be different. For example, the user feature information required to be determined for different modalities can be preset, and correspondingly, the user feature information of the first sensing data in the modality to which it belongs can be determined.
[0104] In one possible scenario, the first sensing data of the at least one modality may include at least one of the following:
[0105] At least one audio data stream;
[0106] At least one video feed;
[0107] At least one physiological signal of the target user.
[0108] This physiological signal may include, but is not limited to, the target user's heart rate and skin conductance information.
[0109] Accordingly, the user characteristic information of the first perceived data includes at least one of the following:
[0110] In response to the first perceived data including audio data, voiceprint features in the audio data are determined;
[0111] In response to the first perceived data, including video data, determine the viewpoint orientation features of the target user as represented by the video data;
[0112] In response to the first sensed data, including physiological signals, at least one of the characteristics of change in the physiological signals and peak values is determined.
[0113] Among them, voiceprint features are used to characterize the source of speech in the audio data in order to identify the speaker.
[0114] The target user's viewpoint direction feature is used to characterize the target user's viewpoint direction (also known as gaze direction or staring direction) when the video data was acquired. For example, the viewpoint direction feature can characterize the direction of the target user's viewing angle in three-dimensional space. Of course, this viewpoint direction feature can also be determined by the content of objects displayed in the video data; for example, the viewpoint direction feature might indicate that the user's viewpoint is on the road surface or pointing towards buildings on either side of the road. This viewpoint direction feature reflects the direction of the user's current gaze and the content the user is viewing.
[0115] Determining the viewpoint orientation characteristics of the target user as represented by the video data can be achieved by analyzing the spatial changes in the scene corresponding to the video data. Alternatively, it can be combined with other video data and historical video data corresponding to this particular video stream to comprehensively determine the user's viewpoint orientation characteristics; there are no specific limitations.
[0116] Among them, the change characteristics of physiological signals can be the rate of change of physiological signals. For example, when the physiological signal is the target user's heart rate, the rate of change can be heart rate variability (HRV), which refers to the small time fluctuations (usually in milliseconds) between successive heartbeat cycles, and characterizes the degree of irregularity of the heartbeat rhythm.
[0117] The peak value of a physiological signal is a rapidly rising sharp peak that appears within the physiological signal. For example, when the physiological signal is a skin conductance signal, the peak value is the skin conductance peak value.
[0118] Of course, there are other possibilities for the first perception data, and correspondingly, there are other possibilities for user characteristic information, without any restrictions.
[0119] S203, based on the user characteristic information of the first perception data, determine the possibility that the first perception data belongs to the valid perception data that the target user expects to collect.
[0120] Since user characteristic information can reflect a user's identity, behavior, or state, it can be used to determine whether the initial perceived data belongs to the valid perceived data that the user intends to collect. Only valid perceived data that the user intends to collect is considered valid data required to perform the target task; otherwise, the initial perceived data may be noise data unrelated to the target task, such as data collected unconsciously by the user or data mixed with environmental background information.
[0121] Among them, when the modality of the first perception data is different, the specific implementation of determining the first perception data as valid perception data based on user feature information will also be different.
[0122] In one possible scenario, if the first perceived data is audio data, the likelihood that the audio data is from the target user can be determined based on the matching degree between the voiceprint features in the audio data and the configured user voiceprint features.
[0123] The configured user voiceprint features are pre-configured voiceprint features of the target user. Therefore, the higher the matching degree between the voiceprint features in the audio data and the configured user voiceprint features, the higher the probability that the audio data is an audio signal emitted by the target user. The higher the probability that the audio data is an audio signal emitted by the target user, the higher the probability that the audio data is the perceptual data that the target user expects to process for the target task.
[0124] In another possible scenario, this application can also obtain second perceptual data required to perform the target task. The second perceptual data includes motion data, which characterizes the motion state of the target electronic device used by the target user. Based on this, if the first perceptual data is video data, the user feature information is the viewpoint orientation feature of the target user represented by the video data. Accordingly, based on the motion vector represented by the motion data and the viewpoint orientation feature, it can be determined whether the video data includes content actively viewed by the target user.
[0125] The target electronic device used by the target user can be the electronic device performing the target task, or it can be one or more other electronic devices besides the target electronic device. For example, the target electronic device can be a wearable device such as a smartwatch or smart glasses, or other types of electronic devices besides the target electronic device.
[0126] The motion vector represented by this motion data can reflect the user's current orientation and direction of movement over a period of time prior. Whether the motion direction represented by this motion vector matches the viewpoint direction feature can indicate whether the video data corresponds to the area the user is actively focusing on.
[0127] For example, if the viewpoint direction represented by the viewpoint direction feature is consistent with the motion direction represented by the motion vector (e.g., the difference is less than a set threshold, which can be considered as consistent direction), it indicates that the target user is looking at an object within their direction of travel, and the video data at this time belongs to the video data collected within the active attention range. If the viewpoint direction is inconsistent with the motion direction, it indicates that the target user's attention has been temporarily attracted by other events to the side or that they have been passively disturbed by objects in the environment, and it is more likely that they have been passively intruded.
[0128] Understandably, video data corresponding to content that the target user actively pays attention to is more valuable to the target task; however, video content that is passively intruded upon may be data that the target user does not care about, and it is not perceptual data required to perform the target task, but rather noise data.
[0129] Of course, in addition to motion data of the target electronic device used by the user, the second sensing data may also include other sensing data unrelated to the target user, such as acceleration data and position data of the target electronic device sensed by the second sensing data, as well as acceleration data, position data and temperature data of other electronic devices besides the target electronic device, etc., without any specific restrictions.
[0130] In another possible scenario, if the first perceived data is a physiological signal, the likelihood that the physiological signal was generated under external stimulation can be determined based on at least one of the physiological signal's variation characteristics and peak values. The higher the likelihood that the physiological signal was generated under external stimulation, the higher the likelihood that the physiological signal is the effective perceived data desired by the target user.
[0131] If the physiological signal is generated without external stimulation, its peak value and rate of change remain consistent with the target user's daily state. Therefore, it cannot demonstrate the influence or role of other sensory data on the execution of the target task, and thus has no particular significance for the task. However, if the physiological signal is generated under external stimulation, then during the execution of the target task, contextual data related to the target task can be analyzed from other sensory data.
[0132] For example:
[0133] When a target user suddenly hears a friend call their name, their skin conductance will show an abnormal peak. In this situation, if a target task arises during the conversation between the target user and their friend, the starting point of the audio data to be analyzed can be determined from the audio data based on the peak value of the target user's skin conductance signal and the corresponding time point. This allows the location where the target user and their friend started their conversation to be determined from the audio data, and the sub-audio data starting from that location can be extracted. This sub-audio data can then be used as context data for the execution of the target task, enabling more effective execution of the target task.
[0134] In this case, even if the video data of the scene where the target user is located is missing during the execution of the target task, the contextual audio data for the execution of the target task can be reasonably extracted from the audio data based on the peak value of the target user's skin conductance signal.
[0135] S204, based on the probability corresponding to the first sensing data, determine the applicability of the first sensing data to the target task.
[0136] The higher the probability that the first perception data is valid perception data that the user expects to collect, the higher its applicability to the target task.
[0137] For example, the probability can be determined as the applicability; or, the applicability can be determined according to the conversion function between the probability and the applicability, without any specific restrictions.
[0138] S205, using at least a portion of the first perception data whose applicability satisfies the target conditions, perform the target task.
[0139] This step can be found in the relevant descriptions of the previous embodiments, and will not be repeated here.
[0140] In this embodiment, user feature information in the first sensing data is determined based on the modality to which the first sensing data belongs. Based on the user feature information, it can be determined whether the first sensing data belongs to the valid sensing data that the user expects to collect. Since the first sensing data that does not belong to the user expects to collect is useless sensing data for performing the target task, the applicability of the first sensing data to the target task is determined based on the validity of the first sensing data. This can filter out sensing data that does not belong to the target user, is not actively concerned by the target user, or cannot reflect the user's intention. In this way, noise data that is irrelevant to the execution of the target task can be filtered out, thereby improving the reliability and accuracy of the target task execution.
[0141] The following example uses target feature information, which includes semantic feature information, in conjunction with... Figure 3 Please provide an explanation. For example... Figure 3 This is another flowchart illustrating the task execution method provided in this application. The method in this embodiment may include:
[0142] S301, determine the target task to be performed and the first perception data of at least one modality required to perform the target task.
[0143] The first perception data is perception data that is associated with the target user.
[0144] S302, For each first perception data, determine the semantic feature information of the first perception data.
[0145] The semantic features of the first perception data are used to characterize the content information expressed by the first perception data.
[0146] For example, if the first perceived data is video data, semantic analysis of each frame can be performed to determine the location, scene, people, and events reflected in the video data, thus obtaining the semantic features represented by the video data. For instance, if the video data is a video of someone operating a computer, semantic analysis can be performed to determine the files and other objects the user is operating on the computer, as well as the specific operations performed on those objects.
[0147] For example, if the first perceived data is audio data, the semantic features of the audio data can be determined by recognizing the semantics expressed in the text. The semantic features of the audio data can reflect information such as the content being discussed, the events, the user's context, and the dialogue scenario (such as a meeting, training session, or conversation).
[0148] For example, the first sensory data is physiological signals. By analyzing changes in these physiological signals, we can determine the current environment and the context of the conversation, thus obtaining semantic feature information. For instance, if the heart rate remains relatively high and stable for an extended period, it indicates that the semantic feature information of the physiological signals represents a running scenario or a scenario involving physiological emotional excitement.
[0149] In this embodiment, the semantic features of the first perceived data can be determined by an intelligent model, or by other means, without any specific limitations.
[0150] S303, determine the degree of matching between the semantic feature information of the first perception data and the semantic feature information of other first perception data.
[0151] Understandably, this matching degree characterizes whether the content information expressed by the first sensing data matches the content information expressed by other first sensing data. Therefore, if the matching degree is too low, it indicates that the content information reflected by the first sensing data differs significantly from that reflected by other first sensing data, suggesting that the first sensing data is likely to be accidentally collected noise data or useless data.
[0152] In one possible implementation, to accurately determine whether the first sensing data is noisy data that is useless to the target task or interferes with it, the common information expressed by the multimodal data can be determined based on the semantic feature information corresponding to the first sensing data and at least one other modality of first sensing data. Here, multimodal refers to the first sensing data of multiple modalities constituted by the first sensing data and at least one other modality of first sensing data. Correspondingly, the matching degree between the information expressed by the first sensing data and the common information can be determined based on the semantic feature information of the first sensing data.
[0153] This common information can be the information that is common to the information expressed by the first perception data of various modalities, that is, the information that is commonly expressed by the first perception data of most modalities.
[0154] For example:
[0155] Suppose we obtain four channels of first-sensory data, each corresponding to one of three modalities. Three of these channels express the information "the target user is in a meeting," while only one channel expresses the information "the target user is running." Therefore, we can determine that the common information is "the target user is in a meeting."
[0156] Based on this, if the information expressed by the first perception data is that the target user is in a meeting, then the first perception data shares the same common information with this event, indicating a high degree of matching. This suggests that the first perception data reflects the same or similar information as other types of first perception data, and therefore, it is highly likely that the first perception data is useful for processing the target task. However, if the information expressed by the first perception data is "the target user is running," then there is a significant deviation between the first perception data and the information expressed by other perception data. In this case, the semantic features expressed by the first perception data are either useless content or introduced noise.
[0157] S304, Based on the matching degree corresponding to the first sensing data, determine the applicability of the first sensing data to the target task.
[0158] The higher the matching degree of the first perception data, the higher its applicability to the target task.
[0159] For example, the matching degree can be determined as the applicability; or, the applicability can be determined according to the conversion function between the matching degree and the applicability, without any specific restrictions.
[0160] S305, using at least a portion of the first perception data whose applicability satisfies the target conditions, perform the target task.
[0161] This step can be referred to in the relevant description of the previous embodiments.
[0162] In this embodiment, the matching degree between the semantic feature information expressed by the first perception data and the semantic feature information expressed by other first perception data can be used to determine whether the first perception data is noisy data, thereby eliminating useless perception data that does not match the semantic features of other first perception data, reducing the interference of useless data on the execution of the target task, and thus improving the reliability and accuracy of the execution of the target task.
[0163] It is understood that the above explanation is based on the example that the target feature information of the first perception data consists of user feature information and semantic feature information. In practical applications, the target feature information may also include both user feature information and semantic feature information. In this case, this application may comprehensively determine the applicability of the first perception data to the target task based on the probability that the first perception data belongs to the effective perception data that the target user expects to collect and the matching degree corresponding to the first perception data.
[0164] For example, the probability and matching degree corresponding to the first perception data are weighted and summed according to the first weight coefficient corresponding to the probability and the second weight coefficient corresponding to the matching degree, and the result of the weighted summation is determined as the applicability of the first perception data to the target task.
[0165] For example, the applicability of the first perception data to the target task can be determined by multiplying the probability of the first perception data with its corresponding matching degree.
[0166] Of course, there are other ways to determine applicability by combining the probability and matching degree of the first perception data, and there are no restrictions on this.
[0167] In the above embodiments of this application, in order to further accurately analyze the applicability of the first sensing data to the target task, this application may further consider the correlation between the target task and the first sensing data. The applicability is then modified or determined based on this correlation.
[0168] The following is combined Figure 4 Please explain this situation. For example... Figure 4 This illustration shows another flowchart of the task execution method provided in this application. The method of this embodiment is applied to an electronic device and may include:
[0169] S401, determine the target task to be performed and the first perception data of at least one modality required to perform the target task.
[0170] The first type of perception data refers to perception data that is associated with the target user.
[0171] S402, Determine the task characteristic information of the target task.
[0172] The task feature information is used to describe the specific task characteristics of the target task.
[0173] For example, the task feature information may include, but is not limited to, one or more of the following: the task type, the task objective, the keywords contained in the target task, and the task content.
[0174] S403, Based on the task feature information of the target task and each first perception data, determine the correlation between the first perception data and the target task.
[0175] The correlation is used to characterize the probability that the first perception data is associated with the task feature information of the target task, and can also reflect the importance of the first perception data when the task type corresponding to the target task is executed.
[0176] For example, the correlation between the first perception data and the target task can be determined based on the task characteristics of the target task and the differences between the first perception data and other first perception data.
[0177] For example, a trained large model or other intelligent model can be used to combine the task feature information of the target task with each first perception data to determine the correlation between each first perception data and the target task.
[0178] In one possible implementation, to more accurately and reasonably determine the relevance between the first perception data and the target task, the scene characteristics of the target user can be determined based on each piece of first perception data. On this basis, the relevance between the first perception data and the target task can be determined based on the task characteristic information of the target task and the scene characteristics. For example, the task characteristic information of the target task, the scene characteristics, and the first perception data can be input into a large-scale intelligent model to obtain the relevance output by the intelligent model.
[0179] These scene features are used to characterize the user's environment, activity, or behavioral state. For example, scene features can characterize whether the target user is in an indoor or outdoor environment, a static state, a walking state, or a moving state, a communication state such as being alone, in a crowd, or having a conversation with multiple people, or a specific work scenario, without any restrictions.
[0180] Determining scene features based on each piece of first-sensory data can be achieved by using at least one feature keyword corresponding to different scene features to determine the scene features corresponding to each piece of first-sensory data. Alternatively, a model can be used to combine each piece of first-sensory data to output the scene features of the target user.
[0181] Furthermore, considering that different users may require different perceptual data when performing the same task, this application can also obtain the target user's personalized information to further improve the accuracy of the determined correlation. This personalized information can record the source and type of perceptual data that the user has accepted when performing different types of tasks in the past. Based on this, the correlation between the first perceptual data and the target task can be determined by combining the target user's personalized information, the task characteristics of the target task, and the scene characteristics.
[0182] S404, determine the target feature information of the first sensing data in its respective modality.
[0183] The target feature information is used to characterize at least one of the semantic features and user features expressed by the first perceived data.
[0184] S405, Based on the target feature information corresponding to the first perception data, determine the applicability of the first perception data to the target task.
[0185] This step can be referred to in the relevant description in any of the previous embodiments, and will not be repeated here.
[0186] It should be noted that in practical applications, steps S402 and S403 can be executed simultaneously with steps S404 and S405, or steps S404 and S405 can be executed first, followed by steps S402 and S403.
[0187] S406, Based on the correlation of the first sensing data, correct the applicability of the first sensing data.
[0188] For example, the applicability can be multiplied by the correlation to obtain the corrected applicability.
[0189] For example, the applicability can be weighted and summed with the relevance to obtain the modified applicability, without any specific restrictions.
[0190] S407, using at least a portion of the first perception data whose applicability satisfies the target conditions, perform the target task.
[0191] This step can be referred to in the relevant description of the previous embodiments, and will not be repeated here.
[0192] In this embodiment, by combining task relevance correction with applicability, the perception data ultimately used to execute the task can be not only related to the user, but also directly related to the current target task, which is conducive to further improving the accuracy of task execution.
[0193] In this embodiment, the example is illustrated by first determining the applicability and then using correlation to correct it. However, it is understood that the corrected applicability of this application can essentially be seen as an applicability determined by combining the target feature information and correlation of the first sensing data. For example, the applicability of the first sensing data is determined by combining at least one of the following: the probability that the first sensing data belongs to the valid sensing data that the target user expects to collect, the matching degree corresponding to the first sensing data, and the correlation.
[0194] To facilitate understanding, the following example uses the combination of the probability, matching degree, and relevance of the first perception data to comprehensively determine its applicability. Figure 5 Please provide an explanation. For example... Figure 5 This illustration shows another flowchart of the task execution method provided in this application. The method of this embodiment can be applied to electronic devices and may include:
[0195] S501, determine the target task to be executed, the task feature information of the target task, and the first perception data of at least one modality required to execute the target task.
[0196] The first perception data is perception data that is associated with the target user.
[0197] S502, based on the first perception data, determine the scene characteristics of the target user.
[0198] S503, for each first perception data, based on the task feature information of the target task and the scene features, determine the correlation between the first perception data and the target task.
[0199] S504, for each first perception data, determine the user feature information and semantic feature information of the first perception data in its respective modality.
[0200] S505, based on the user characteristic information of the first perception data, determine the possibility that the first perception data belongs to the valid perception data that the target user expects to collect.
[0201] It should be noted that in practical applications, the execution order of operations for determining probability and correlation can be interchanged or performed synchronously, without any restrictions.
[0202] S506, based on the semantic feature information of the first perceptual data and the semantic feature information of the first perceptual data of at least one other modality, determine the common information of the multimodal common expression.
[0203] For details on how to determine common information, please refer to the relevant descriptions in the preceding embodiments.
[0204] Specifically, in practical applications, the semantic feature information corresponding to the first sensing data and at least one other modality's first sensing data may not uniquely determine a common item information, i.e., at least two candidate common item information may be identified. Among these, the number of first sensing data corresponding to different candidate common item information is the same. Therefore, this application can also determine the target first sensing data based on the correlation and probability corresponding to each first sensing data, where the product of the correlation and probability corresponding to the target first sensing data is the highest. Based on this, the candidate common items corresponding to the target first sensing data can be determined as common item information.
[0205] For example, suppose there are four modalities of first-sensory data. Two modalities determine event information 1, and the other two modalities determine event information 2. Event information 1 and event information 2 are contradictory conclusions or completely different event information. For instance, combining audio with first-sensory data from another modality determines event information 1 as the user being calm, while based on physiological signals and first-sensory data from another modality, it determines the user being excited. In this case, this application can calculate the product of the probability and relevance corresponding to each type of first-sensory data, and determine the event information corresponding to the first-sensory data with the highest product as the common event information.
[0206] In practical applications, if multiple candidate common items of information appear, each candidate common item of information can be treated as a common item of information; or, the matching degree of each first perception data corresponding to the candidate common item of information can be set as the first matching degree, while the matching degree of the first perception data that does not correspond to any candidate common item of information can be set as the second matching degree, and the second matching degree is lower than the first matching degree.
[0207] Of course, if there are two candidate common items, after determining the final common items, the matching degree of each subsequent first perception data can be multiplied by a discount factor to appropriately reduce the value of the matching degree, and each first perception data and the corresponding candidate common items information can be recorded for subsequent learning.
[0208] S507, Based on the semantic feature information of the first perception data, determine the matching degree between the item information expressed by the first perception data and the common item information.
[0209] It should be noted that the order in which the relevance, probability, and matching degree of the first perception data are determined is not limited to... Figure 5 As shown, in practical applications, the order of determining relevance, probability, and matching degree can be interchanged or performed simultaneously.
[0210] S508, based on the probability, matching degree and relevance of the first sensing data, determine the applicability of the first sensing data to the target task.
[0211] In one example, the applicability score of the first perception data to the target task can be expressed as: ;
[0212] Where R represents the probability corresponding to the first perceived data, A represents the matching degree corresponding to the first perceived data, and W represents the relevance corresponding to the first perceived data. It can be seen that the applicability determined in this way can also be considered as an initial applicability determined based on the matching degree and probability of the first perceived data. Then, the initial fitness score is corrected using the correlation W to obtain the corrected fitness score: .
[0213] In another example, the applicability of the target task can be determined by weighted summation of the probability, matching degree, and relevance of the initial perceived data. This approach can also be viewed as follows: first, weighted summation of probability and matching degree is performed to obtain an initial applicability; then, the product of relevance and weight is added to this initial applicability to correct it and obtain the final applicability.
[0214] S509, if the applicability of the first perception data is not lower than the first threshold, the first perception data is determined as the target perception data required to process the target task.
[0215] S510, if the applicability of the first sensing data is not lower than the second threshold and is lower than the first threshold, perform data compression processing on the first sensing data, and determine the first sensing data obtained by compression processing as the target sensing data required to process the target task.
[0216] The first threshold is greater than the second threshold.
[0217] S511, if the applicability of the first perception data is lower than the second threshold, the first perception data shall not be used as the target perception data for processing the target task.
[0218] S512, based on the determined target perception data, execute the target task.
[0219] This step can be referred to in the relevant description of the previous embodiments, and will not be repeated here.
[0220] In the above embodiments of this application, after the execution of the target task is completed, this application may also output the reason information for each first perception data being blocked or determined as target perception data for executing the target task, so as to store it as context data corresponding to the target task, so that the user can understand the relevant reason information and adjust the first perception data expected to be used for the corresponding task as needed.
[0221] In one approach, the user can adjust the first perception data that the task expects to use by adjusting only the importance of the first perception data from various sources to the task, so that when selecting the first perception data later, the perception data required to perform the task can be filtered based on the importance of the first perception data from each source set by the user.
[0222] In the case where the user adjusts the content type or modality of the first perceptual data used for the desired target task, this application can record the user's adjusted task and perceptual data preference information into the user's personalized information so that before executing other tasks, the applicability of the relevant perceptual data can be adjusted in combination with the user's personalized information, or the weight of the probability and matching degree of the relevant perceptual data can be adjusted.
[0223] This personalized information can also record the generation time of each preference information entry, allowing for timely determination of its influence weight within the personalized information based on its generation time. As the time elapsed since the preference information was generated increases, its influence weight gradually decreases.
[0224] In another approach, users can configure whether first-sensory data from various sources is used for a certain type of task, generating decision configuration information and setting this decision configuration information to be valid only once or persistently. In this case, if the user configures first-sensory data from a certain source not to be used to perform a certain type of task, then when that type of task is to be performed subsequently, the previous approach can be bypassed, and the first-sensory data from that source can be directly masked based on the user-configured decision configuration information.
[0225] In any of the above embodiments of this application, the sensors on the electronic device or other electronic device typically continuously collect first sensing data, and the first sensing data from different sources may be out of sync. Moreover, performing the target task usually only requires first sensing data within a certain period of time, and to facilitate data filtering, only the first sensing data within a certain period of time is processed each time.
[0226] Based on this, the electronic device can obtain candidate first sensing data for at least one modality. These candidate first sensing data can be first sensing data within a set time period (e.g., 30 minutes) prior to the current moment. Furthermore, the candidate first sensing data can be aligned based on the timestamps of each modality. Then, first sensing data is determined from each candidate first sensing data according to a set window, resulting in first sensing data for at least one modality. The length of the set window can be 50 seconds or 1 minute, etc., without specific limitations. Correspondingly, after each determination of first sensing data for at least one modality, the scheme of this application can be used to process each stream of first sensing data separately.
[0227] To facilitate a clear and intuitive understanding of the solution presented in this application, the following example demonstrates how to obtain perception data in three modalities. Figure 6 A brief explanation is provided.
[0228] exist Figure 6 The following example illustrates how to obtain video data, audio data, and physiological signals associated with a target user.
[0229] First, after time-aligning the video data, audio data, and physiological signals, a set window is used to segment the sensing data into sensing data segments. Each time, a set of time-aligned sensing data segments is scored. Each set of sensing data segments includes a video data segment, an audio data segment, and a physiological signal segment.
[0230] The sensors for each sensing data stream can operate continuously to collect sensing data. Alternatively, this application can also activate the image sensor to collect image data only when a change in the scene in which the target user is located is detected, in order to reduce power consumption.
[0231] Secondly, for each modality's perceptual data segment (i.e., the first perceptual data), by combining these segments with other types of perceptual data, the user's scenario type can be identified. Combining the scenario type, the task characteristic information of the target task to be executed, and the target user's personalized information, the correlation W between each perceptual data segment and the target task is determined. The correlation W can range from 0 to 1.
[0232] Furthermore, this application will determine the probability R of each perceived data segment belonging to the perceived data that the user expects to collect, specifically as follows:
[0233] Determine whether the voiceprint of the audio data segment is the voiceprint of the target user, and determine the probability R that the audio data segment is audio sent by the target user based on the determination result;
[0234] Based on the viewpoint direction in the video data segment, the probability R of the video segment being content actively viewed by the target user can be determined;
[0235] Based on the rate of change of the physiological signal center rate and the peak value of skin conductance, the probability R of the physiological signal being generated under external stimulation is determined.
[0236] The value of R can also be in the range of 0 to 1.
[0237] Then, based on the semantic features of "in a meeting" represented by video data, the semantic features of "in a meeting" represented by audio data, and the semantic features of "running" represented by physiological modality, the common information of these multiple modalities is determined to be "in a meeting". The matching degree (i.e., modal consistency) A between the information of the matter expressed by the semantic features of each perceptual data and this common information is determined.
[0238] Based on this, the applicability score of the perceptual data segment for each modality to the target task can be expressed as: .
[0239] Next, the applicability of the perceived data segment is compared with the set first threshold 'a' and second threshold 'b' to decide whether to retain the perceived data segment, and whether to retain all or part of the perceived data segment. Here, 'a' is greater than 'b'.
[0240] For example, if the score is not less than a, the entire sensing data segment is retained; if a > Score > b, the data segment is compressed or downsampled and used as the retained sensing data segment; if the score is less than or equal to b, the sensing data segment is blocked.
[0241] After this, the target task can be executed based on the final retained perception data segments.
[0242] Finally, after completing the target task, information about each perception data segment can be output to allow users to adjust or trigger updates to their personalized information. For details, please refer to the previous introduction, which will not be repeated here.
[0243] It should be noted that the voiceprint features, fingerprint features and other identity recognition data involved in this application shall be obtained through legal means with the authorization or consent of the relevant users, and stored and processed in accordance with applicable data protection requirements.
[0244] This application also provides an electronic device in its embodiments. For example... Figure 7 As shown, a schematic diagram of the composition structure of the electronic device is presented, which includes at least a processor 701 and a memory 702.
[0245] The processor 701 is configured to determine the target task to be executed and at least one modality of first perception data required to execute the target task, wherein the first perception data is perception data associated with the target user; determine the target feature information of the first perception data in its modality, wherein the target feature information is used to characterize at least one of the semantic features and user features expressed by the first perception data; determine the applicability of the first perception data to the target task based on the target feature information corresponding to the first perception data; and execute the target task using at least a portion of the first perception data whose applicability satisfies the target conditions.
[0246] The memory 702 is used to store the programs required for the processor to perform operations.
[0247] For details regarding the specific operations performed by the processor, please refer to the relevant descriptions in the preceding embodiments, which will not be repeated here.
[0248] It is understood that the electronic device may also include a display unit 703 and an input unit 704.
[0249] Of course, the electronic device can also have more than Figure 7 There are no restrictions on the number of components, whether more or fewer.
[0250] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the task execution methods provided in this application.
[0251] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the task execution methods provided in this application.
[0252] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.
[0253] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence or the part that has been contributed, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0254] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.
[0255] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).
Claims
1. A task execution method, comprising: The target task to be performed and the first perception data of at least one modality required to perform the target task are determined, wherein the first perception data is perception data associated with the target user; Determine the target feature information of the first perceived data in its modality, wherein the target feature information is used to characterize at least one of the semantic features and user features expressed by the first perceived data; Based on the target feature information corresponding to the first sensing data, the applicability of the first sensing data to the target task is determined; The target task is performed using at least a portion of the first perception data in which the applicability satisfies the target conditions.
2. The task execution method according to claim 1, wherein the target feature information includes: User characteristic information; The step of determining the applicability of the first sensing data to the target task based on the target feature information corresponding to the first sensing data includes: Based on the user characteristic information of the first sensing data, determine the possibility that the first sensing data belongs to the valid sensing data that the target user expects to collect. Based on the probability corresponding to the first sensing data, the applicability of the first sensing data to the target task is determined.
3. The task execution method according to claim 1, wherein the target feature information includes: Semantic feature information; The step of determining the applicability of the first sensing data to the target task based on the target feature information corresponding to the first sensing data includes: Determine the degree of matching between the semantic feature information of the first sensing data and the semantic feature information of other first sensing data; Based on the matching degree corresponding to the first sensing data, the applicability of the first sensing data to the target task is determined.
4. The task execution method according to any one of claims 1 to 3, further comprising: Determine the task characteristic information of the target task; Based on the task feature information of the target task and each of the first perception data, the correlation between the first perception data and the target task is determined; Based on the correlation of the first sensing data, the applicability of the first sensing data is corrected.
5. The task execution method according to claim 4, wherein determining the correlation between the first sensing data and the target task based on the task feature information of the target task and each of the first sensing data includes: Based on the first perception data, the scene characteristics of the target user are determined; Based on the task feature information of the target task and the scene features, the correlation between the first perception data and the target task is determined.
6. The task execution method according to claim 3, wherein determining the matching degree between the semantic feature information of the first perceived data and the semantic feature information of other first perceived data includes: Based on the semantic feature information of the first perceptual data and at least one other modality of first perceptual data, common information of items jointly expressed by multiple modalities is determined; Based on the semantic feature information of the first perceived data, the matching degree between the item information expressed by the first perceived data and the common item information is determined.
7. The task execution method according to claim 2, wherein the first perception data of the at least one modality includes at least one of the following: At least one audio data stream; At least one video feed; At least one physiological signal of the target user; Determining the user characteristic information of the first perceived data includes at least one of the following: In response to the first perceived data including audio data, voiceprint features in the audio data are determined; In response to the first perceived data including video data, the viewpoint orientation features of the target user represented by the video data are determined; In response to the first sensed data including the physiological signal, at least one of the variation characteristics and peak values of the physiological signal is determined.
8. The task execution method according to claim 7, based on the user feature information of the first perceived data, determining the probability that the first perceived data belongs to the valid perceived data that the target user expects to collect, includes at least one of the following: In response to the first perceived data being audio data, based on the matching degree between the voiceprint features in the audio data and the configured user voiceprint features, the possibility that the audio data is audio data emitted by the target user is determined; In response to the first perceived data being video data, based on the motion vectors represented by the motion data and the viewpoint direction features, it is determined whether the video data includes content actively viewed by the target user; wherein... The motion data is the second perceptual data required to perform the target task, and the motion data is used to characterize the motion state of the target electronic device used by the target user. In response to the first perceived data being the physiological signal, based on at least one of the variation characteristics and peak values of the physiological signal, the probability that the physiological signal is a physiological signal generated under external stimulation is determined.
9. The task execution method according to claim 1, wherein executing the target task using at least a portion of the first perceptual data in which the applicability satisfies the target condition comprises: If the applicability of the first sensing data is not lower than the first threshold, the first sensing data is determined as the target sensing data required to process the target task; If the applicability of the first sensing data is not lower than the second threshold and is lower than the first threshold, the first sensing data is compressed, and the first sensing data obtained by the compression is determined as the target sensing data required to process the target task, wherein the first threshold is greater than the second threshold. Based on the determined target perception data, the target task is executed.
10. An electronic device, comprising: Processor and memory; The processor is configured to: determine a target task to be executed and first perceptual data of at least one modality required to execute the target task, wherein the first perceptual data is perceptual data associated with a target user; determine target feature information of the first perceptual data in its modality, wherein the target feature information is used to characterize at least one of semantic features and user features expressed by the first perceptual data; determine the applicability of the first perceptual data to the target task based on the target feature information corresponding to the first perceptual data; and execute the target task using at least a portion of the first perceptual data whose applicability satisfies the target conditions. The memory is used to store the programs required by the processor to perform operations.
