Data processing method and device

Abstract

The invention discloses a data processing method and device, and belongs to the technical field of data processing, and the method comprises the steps: obtaining multi-modal behavior data; determining behavior description data of the user behavior according to the multi-modal behavior data; screening first multi-modal behavior data associated with the behavior description data from the multi-modal behavior data; and generating user memory data corresponding to the user behavior based on the behavior description data and the first multi-modal behavior data.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and specifically relates to a data processing method and a device thereof. Background Art

[0002] With the rapid development of electronic device technology and mobile Internet, more and more users can manually edit information about important life experiences such as gatherings with friends or family, traveling, and watching performances through various applications in electronic devices, hoping that when they look back one day in the future, they can recall the details from past records.

[0003] However, the information of these important life experiences covers a wide range and is diverse. For example, when traveling by car with friends, the information that the user wants to edit may include images and videos taken during the travel, the route of the driving, the attractions visited, the food eaten, the performances watched, the songs listened to while driving, etc. If the user edits this information manually, it will increase the complexity of information editing and reduce the efficiency of information editing. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a data processing method, device, electronic device and storage medium, which can reduce the complexity of information editing and improve the efficiency of information editing.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0006] Acquire multimodal behavioral data;

[0007] Determine behavior description data of user behavior based on multimodal behavior data;

[0008] Filtering first multimodal behavior data associated with the behavior description data from the multimodal behavior data;

[0009] Based on the behavior description data and the first multimodal behavior data, user memory data corresponding to the user behavior is generated.

[0010] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0011] An acquisition module, used to acquire multimodal behavior data;

[0012] A determination module, used to determine behavior description data of user behavior based on the multimodal behavior data;

[0013] A screening module, configured to screen first multimodal behavior data associated with the behavior description data from the multimodal behavior data;

[0014] A generation module is used to generate user memory data corresponding to the user behavior based on the behavior description data and the first multimodal behavior data.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, the steps of the data processing method shown in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the data processing method shown in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a display interface, the display interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the data processing method shown in the first aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the data processing method shown in the first aspect.

[0019] In the embodiment of the present application, the behavior description data of the user's behavior is determined through the user's multimodal behavior data, and the first multimodal behavior data associated with the behavior description data is screened from the multimodal behavior data, and then, based on the behavior description data and the first multimodal behavior data, the user memory data corresponding to the user behavior is generated. In this way, by collecting multimodal behavior data, the types of data elements can be enriched, the comprehensiveness and completeness of the data obtained from user behavior can be improved, and the accuracy of determining the source of user behavior data can be improved, and the first multimodal behavior data can be summarized and sorted according to the behavior description data, without the need for the user to manually edit the information of the user's life experience, thereby reducing the complexity of information editing and improving the efficiency of information editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a data processing method provided for some embodiments of the present application;

[0021] Figure 2 A schematic diagram of a flow chart of image data processing of a data processing method provided in some embodiments of the present application;

[0022] Figure 3 A schematic diagram of a process for processing multimodal behavior data in a data processing method provided in some embodiments of the present application;

[0023] Figure 4A schematic diagram of the structure of a data processing device provided for some embodiments of the present application;

[0024] Figure 5 A schematic diagram of the structure of an electronic device provided for some embodiments of the present application;

[0025] Figure 6 A schematic diagram of the hardware structure of an electronic device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] Friends or family gatherings, travel, important performances, etc. are important life experiences for users. Most users hope to remember these important life experiences and hope to recall the details and feelings of the situation from past records when they look back one day in the future. For example, when traveling by car with friends, the information about life experiences that users want to record may include photos taken during the trip, the route of the driving, the attractions visited, the food eaten, the performances watched, and the playlists listened to while driving.

[0029] In the related art, the recall image function of the album application can be used, that is, the images taken or videos recorded by the user in a certain place can be summarized for loop playback; or some map navigation applications can provide a light up corners of the earth function, that is, when the user uses the map navigation application for navigation, the user can light up the corresponding location on the city passed by, thereby recording his or her footprints.

[0030] However, the dimensions of the information on the user's life experience obtained by the above two methods are relatively single, and only single-dimensional data can be organized into memoirs, and they cannot be directly associated with each other, so that users need to repeatedly adjust between multiple applications when viewing this information, which increases the user's operation to view this information. In addition, compared with the user's entire journey experience, a large amount of data in other dimensions is still missing, such as food, attractions, etc., which requires users to intervene and spend a lot of energy on secondary processing or re-editing, increasing the complexity of information editing and reducing the efficiency of information editing.

[0031] In order to solve the problems in the related art, the embodiments of the present application provide a data processing method, device and electronic device. Figures 1 to 6 , the data processing method provided in the embodiment of the present application is described in detail through specific embodiments and their application scenarios.

[0032] First, combine Figure 1 A data processing method provided in an embodiment of the present application is described in detail.

[0033] Figure 1 A flowchart of a data processing method provided for some embodiments of the present application.

[0034] like Figure 1 As shown, the data processing method provided in the embodiment of the present application can be applied to electronic devices. Based on this, the data processing method can include steps 110 and 140 as shown below.

[0035] Step 110, acquiring multimodal behavior data; Step 120, determining behavior description data of the user behavior based on the multimodal behavior data; Step 130, screening first multimodal behavior data associated with the behavior description data from the multimodal behavior data; Step 140, generating user memory data corresponding to the user behavior based on the behavior description data and the first multimodal behavior data.

[0036] Therefore, by collecting multimodal behavior data, the types of data elements can be enriched, the comprehensiveness and completeness of the data obtained on user behavior can be improved, and the accuracy of determining the source of user behavior data can be improved. In addition, the first multimodal behavior data can be summarized and organized according to the behavior description data, eliminating the need for users to manually edit information on user life experiences, thereby reducing the complexity of information editing and improving information editing efficiency.

[0037] The above steps are described in detail below, as shown below.

[0038] Involving step 110, the multimodal behavior data in the embodiment of the present application may include, but is not limited to, data generated when a user operates on an application, applet, web page, or other platform in an electronic device, including browsing, clicking, sliding, long pressing, and other behavior data, that is, data that is fully perceived in the user's electronic device, and may specifically include at least one of the following data types: image data, audio data, text data, and numerical data.

[0039] Among them, image data may include but is not limited to images or videos in album applications, images and videos collected in applications of electronic devices (such as instant messaging applications, shopping applications), reference images in the image editing process, processing parameters, etc.

[0040] Audio data may include, but is not limited to, audio in an album application, recordings, local music, downloaded audio, and audio played by a music player application.

[0041] Text data may include, but is not limited to, text information viewed and edited by a user in an electronic device, such as text message content, schedules, notebooks, and favorites links.

[0042] Numerical data includes, but is not limited to, numerical data provided to users by the operating system of electronic devices and numerical data provided by various applications in electronic devices based on user behavior. Among them, the numerical data provided to users by the operating system include the battery level of the mobile phone, weather conditions, calendar, and alarm clock. The numerical data provided by various applications based on user behavior include the geographical location and received text messages.

[0043] It should be noted that the embodiments of the present application can obtain multimodal behavior data of these data types by performing data embedding on electronic devices, thereby recording the user's behavioral actions over a period of time.

[0044] Therefore, the embodiments of the present application can integrate various types of data generated by electronic devices, including four types of data such as images, audio, text, and numerical data, thereby solving the problem of single integration elements and lack of information sources for determining user behavior in the prior art.

[0045] Regarding step 120 , in some embodiments of the present application, step 120 may specifically include step 1201 and step 1202 .

[0046] Step 1201, determining user behavior based on multimodal behavior data, the user behavior includes at least one of the following: a first user behavior corresponding to user experience time, a second user behavior corresponding to user experience location, and a third user behavior corresponding to user experience event; wherein the user experience time includes at least one of the following: anniversaries marked by the user, holidays provided by the device system, public holidays provided by the device system, and solar terms provided by the device system; the experience location includes at least one of the following: a temporary location of the user, a commemorative location marked by the user; the user experience event includes at least one of the following: a non-routine event, a planned event.

[0047] The anniversaries marked by the user may include but are not limited to birthdays, wedding anniversaries, death anniversaries, etc. Holidays may include but are not limited to Chinese and foreign festivals and holidays. The memorial places marked by the user may include but are not limited to places the user has visited, places the user plans to visit, places the user likes or dislikes, and places involved in images, texts, or web pages browsed by the user.

[0048] Step 1202: Generate behavior description data of the user behavior according to the user behavior.

[0049] Therefore, by collecting and integrating multimodal behavior data of various data types in different scenarios, the problem of incomplete sources of user behavior data in the prior art can be solved, and then the accuracy of determining user behavior can be improved by analyzing complete and comprehensive multimodal behavior data.

[0050] In an embodiment of the present application, the multimodal behavior data involved in the above step 1201 may include multiple contents, and the determination of user behavior may also include multiple contents. Based on this, an embodiment of the present application provides the following method to determine user behavior, as shown below.

[0051] In some embodiments of the present application, the first user behavior may be determined based on user experience time.

[0052] Based on this, since the multimodal behavior data in the embodiments of the present application may include at least one of the following user experience time: anniversaries marked by the user, holidays provided by the device system, and public holidays provided by the device system. Therefore, the user behavior can be determined based on the user's behavior during the user's experience time. Exemplarily, on the user's birthday, the user's behavior on the birthday can be taken as the first user behavior, where the user may arrive at the user's permanent location and perform routine actions, that is, go to work and get off work at the company as usual, or go to the user's temporary location and perform non-routine actions.

[0053] In some embodiments of the present application, the second user behavior may be determined based on a location experienced by the user.

[0054] Based on this, the multimodal behavior data includes location data of the user's location within N time periods, the location data includes location data, location type and user stay time, the second user behavior includes the user's behavior at the user's temporary location, N is an integer greater than 1, based on this, step 1201 can specifically include steps 12011 to 12013.

[0055] Step 12011, based on the location data of the user's location in N time periods, determine the user's permanent location and the user's movement trajectory in each of the N time periods.

[0056] Specifically, the step 12011 may include steps 120111 to 120113.

[0057] Step 120111, based on the location data of the user's location in N time periods, the location in each time period is fitted by a density clustering algorithm to obtain the user's stay location in each time period.

[0058] The location data in the embodiments of the present application include but are not limited to satellite positioning location data, Bluetooth positioning location data, Wi-Fi positioning location data, and base station positioning location data. The density clustering algorithm includes but is not limited to the density-based spatial clustering of applications with noise (DBSCAN).

[0059] Exemplarily, according to the positioning function of the electronic device, the location data of the positioning locations are collected at preset intervals. Since these positioning locations are discrete, the same location may have multiple location data. Therefore, DBSCAN can be used to fit the positioning locations within each time period based on the location data of the positioning locations within the time period, and the user's stay locations within the time period can be obtained, such as user stay location A, user stay location B, and user stay location C.

[0060] Step 120112, based on the user's stay location in each time period, generate the user movement trajectory for each time period.

[0061] Exemplarily, if the user's staying locations within the time period include user staying location A, user staying location B, and user staying location C, then user staying location A, user staying location B, and user staying location C can be sorted and recorded in the order in which these locations were collected to obtain the user's movement trajectory for the time period.

[0062] Here, it should be noted that since the user's stay location is obtained by fitting the location data of at least one positioning location, if the user's stay location is obtained by the location data of one positioning location, then the time when the user's stay location is collected can be the time when the location data of the positioning location is collected. If the user's stay location is obtained by the location data of at least two positioning locations, then the time when the user's stay location is collected can be the time when the location data of the positioning location is collected earliest among the times when the location data of the at least two positioning locations are collected.

[0063] Step 120113, filter the user's permanent location from the user's stay locations, wherein the user's permanent location satisfies at least one of the following conditions: the number of times the location data of the user's permanent location appears in N user movement trajectories is greater than or equal to a preset number of times, the location type of the user's permanent location matches the preset location type, and the user's stay time at the user's permanent location matches the preset stay time.

[0064] Among them, the preset location types in the embodiments of the present application include but are not limited to: industrial sites, commercial sites, residential sites, public facility sites, sports venues, and agricultural land sites.

[0065] For example, if N is 10, and the number of times the user stay location A appears in 10 user movement trajectories is 10, which is greater than the preset number of times 5, then the user stay location A can be considered as the user's permanent location. Also, if the location type of the user's permanent location is residential and matches the preset point type, such as residential, then the user stay location A can be considered as the user's permanent location. Also, if the user's user stay time at the user stay location A is 8 hours and matches the preset stay time, such as 5-24 hours, then the user stay location A can be considered as the user's permanent location.

[0066] Step 12012: based on the user's permanent location, filter out the user's temporary location from the user's movement trajectory in each time period.

[0067] For example, the user's stay location in the fitted user movement trajectory may be compared with the user's permanent location to obtain the user's temporary location.

[0068] Furthermore, the user's stay locations in the fitted user movement trajectory can be compared with the user's permanent locations to obtain non-daily stay points, and then, based on the location data, location type and user stay time of the non-daily stay points, the user's temporary stay locations can be further screened from the non-daily stay points.

[0069] For example, when the user stays at the non-routine stop point for 24 hours, the user's temporary stop location can be determined.

[0070] Step 12013, determining the behavior of the user at the user's temporary location as the second user behavior.

[0071] Therefore, by defining the user's permanent location and the user's temporary location and distinguishing between the two, the irregular and occasional user's temporary location behavior is determined as the second user behavior.

[0072] In some embodiments of the present application, the third user behavior may be determined based on the user experience event. In this embodiment, two methods are provided based on different attributes of the user experience event, as shown below.

[0073] In some instances, the third user behavior includes behavior corresponding to non-routine events. Based on this, step 1201 may specifically include steps 12014 to 12017.

[0074] Step 12014, based on the multimodal behavior data, determine the user's daily behavior data through a frequent mining algorithm, and the user's daily behavior data includes the user's regular or frequent behavior data.

[0075] Step 12015, based on the user's daily behavior data, determine the user's non-daily behavior data from the multimodal behavior data.

[0076] Step 12016, perform data analysis on the user's non-routine behavior data to obtain non-routine events.

[0077] Exemplarily, behavioral data that occurs regularly or frequently is defined as daily events. Based on this, multimodal behavioral data can include usage data of various applications in electronic devices. Frequent patterns of applications can be mined through traditional machine algorithms such as association rule mining algorithms (Apriori) and decision trees. For example, a user usually has the habit of listening to songs on an audio player. Usually, he may listen to many songs, but one day he may listen to sad songs all day, and the number of times and duration of listening to songs are much longer than usual. Therefore, it can be inferred that the user may have been heartbroken at that time, and it can be regarded as a non-daily event.

[0078] Step 12017, determining the behavior corresponding to the non-routine event as the third user behavior.

[0079] Exemplarily, the example in step 12016 can still be used for explanation, and the behavior corresponding to the non-daily event of a broken heart, namely, the behavior of listening to sad love songs, can be determined as the third user behavior.

[0080] Therefore, by defining daily events and non-daily events, the two are distinguished, and irregular and occasional events are marked as the third user behavior.

[0081] In other instances, the third user behavior includes behavior corresponding to a planned event, based on which, step 1201 may specifically include steps 12018 to 12020.

[0082] Step 12018, filtering data of user pre-planned events from the multimodal behavior data.

[0083] Step 12019, perform data analysis on the data of the user's pre-planned events to obtain planned events.

[0084] Step 12020, determining the behavior corresponding to the planned event as the third user behavior.

[0085] For example, if the user's relevant ticketing data is identified, that is, the user has purchased tickets for a singer's concert on October 10, booked a flight ticket to place A on October 1, etc., then it can be determined that the user has a planned event, that is, a planned event for watching a performance, and a planned event for traveling. At this time, the behavior of purchasing concert surroundings corresponding to the planned event for watching a performance can be determined as the third user behavior, or the behavior of booking accommodation in place A, checking the route of a scenic spot, etc. corresponding to the planned event for traveling can be determined as the third user behavior.

[0086] Involving step 130, since the multimodal behavior data in the embodiment of the present application includes at least one of the following data types: image data, audio data, text data, and numerical data, the acquired multimodal behavior data can be screened and preprocessed to remove irrelevant information and retain content related to the user's important memories, as shown below.

[0087] Based on this, step 130 may specifically include step 1301 and step 1302. This step includes data cleaning, deduplication, format conversion, etc. to ensure data quality.

[0088] Step 1301: Process the data corresponding to each type in the multimodal behavior data through the encoder corresponding to each data type to obtain user data associated with the behavior description data under each type. Step 1302: Associate the user data associated with the behavior description data under each type through the large model to obtain the first multimodal behavior data.

[0089] Exemplarily, the basic data is converted into a digital matrix through an encoder, and then connected to the embedding layer space of the large model as supplementary input information for the large model. For the four data types, there are four data encoders. The visual data encoder converts image pixels into image embedding data, the audio data encoder converts audio signals into audio embedding data, the text data encoder converts multilingual text into text embedding data, and the numerical embedding data encoder concatenates numerical embedding data into numerical embedding data.

[0090] Therefore, user behavior data is automatically organized and aggregated based on the big model, and important elements are extracted based on intent understanding and sentiment analysis, thus solving the problem that existing technologies lack sentiment analysis and users have difficulty in empathy.

[0091] Here, in the embodiment of the present application, the data types include multiple types, so for different data types, the following method of determining the user data associated with the behavior description data under each type is provided.

[0092] Based on this, in some embodiments, the encoder includes a visual data encoder, the user data includes image embedding data, and step 1301 may specifically include steps 13011 to 13014.

[0093] Step 13011, using a visual data encoder, matches the aspect ratio of the image data with a preset aspect ratio to obtain a target preset aspect ratio that matches the image data.

[0094] For example, Figure 2 As shown, a dynamic aspect ratio matching (Dynamic Aspect Ratio Matching) processing step can be performed, that is, the visual data encoder dynamically matches the optimal aspect ratio from a predefined aspect ratio set to maintain the natural aspect ratio of the image, wherein the aspect ratio set may include aspect ratios such as 1:1, 1:2, and 2:1, and the visual data encoder matches the aspect ratio of the input image with these predefined aspect ratios to obtain a target preset aspect ratio that matches the image data.

[0095] Step 13012: perform image segmentation processing on the image corresponding to the image data according to the target preset aspect ratio to obtain at least two image segments.

[0096] Exemplarily, an image division & thumbnail processing step may be performed, that is, after determining a suitable aspect ratio, the image is adjusted to a corresponding resolution and then divided into at least two image segments of 448×448 pixels.

[0097] In addition, to capture the global context, the model also generates a thumbnail of the entire image, which is also scaled to 448×448 pixels.

[0098] Step 13013: perform pixel shuffling processing on at least two image segments to obtain at least two rearranged image segments, wherein the resolution of the rearranged image segments is greater than the resolution of the image.

[0099] Thus, a pixel shuffle process can be performed, which is an operation used to reduce the number of visual tokens. Through this operation, the number of visual tokens represented by the image can be reduced to one-fourth of the original number, which helps to improve the computational efficiency of the model when processing high-resolution images.

[0100] Step 13014: adjust the data dimensions of at least two rearranged image segments to the preset input data dimensions of the large model to obtain image embedding data.

[0101] Exemplarily, at least two rearranged image segments may be passed through a fully connected layer (MLP Projector) to align the dimension of the image data output to the 256 tokens input to the large model, thereby obtaining image embedding data that can be input to the large model.

[0102] In some other embodiments, the encoder includes an audio data encoder, the user data includes audio embedded data, and step 1301 may specifically include steps 13015 to 13017.

[0103] Step 13015, extracting user audio features from the audio waveform of the audio data.

[0104] For example, features can be extracted from the original audio waveform, and common features include Mel-Frequency Cepstrum Coefficients (MFCCs), etc. These features can capture user audio features such as the tone and speech speed of the audio.

[0105] Step 13016, convert the user audio features into audio conversion data through a small convolutional neural network, and the feature dimension of the audio conversion data is smaller than the feature dimension of the audio data.

[0106] Exemplarily, features are converted into embedding sequences through a small convolutional neural network (SCNN), which have lower dimensions and can capture important information of the audio, and the audio is subsampled to reduce the sequence length until the last convolutional layer outputs a vector of a certain dimension as an embedding.

[0107] Step 13017, discretize the audio conversion data through a multi-scale residual vectorization algorithm to obtain audio embedding data.

[0108] For example, the embedded sequence may be converted into a discrete token sequence using a residual vector quantization technique through a multi-scale residual vectorization algorithm (RVQ).

[0109] In this way, by encoding information of different granularities at different layers through the RVQ algorithm, integrity and compactness are balanced, and the accuracy of obtaining audio embedding data is improved.

[0110] In some other embodiments, the encoder includes a numerical data encoder, the user data includes numerical embedded data, and step 1301 may specifically include steps 13018 to 13020.

[0111] Step 13018, using a numerical data encoder, adjusts the numerical data according to preset standardized quantitative parameters to obtain standardized quantitative numerical data.

[0112] For example, the numerical data encoder can be used to preprocess and standardize the numerical data, that is, standardize the data such as the latitude and longitude of the geographical location, the temperature and humidity of the weather conditions to a unified scale, such as using Z-score standardization or minimum-maximum standardization. One-hot encoding is used to convert the numerical data into quantitative data.

[0113] Step 13019, based on the data type of the numerical data, extract scene feature data corresponding to the data type from the standardized quantitative numerical data.

[0114] For example, key features can be selected based on the characteristics and importance of the data, such as the latitude and longitude of the geographic location, the average temperature, humidity, and precipitation probability of the weather. New features can be constructed based on data scenario understanding, such as extracting the places frequently visited by users from geographic location data, or extracting trend and seasonal features from time series data. For time series data, timestamps are converted into time features, such as hours, weeks, months, etc., and their relevance to specific events.

[0115] Step 13020, reconstructing the scene feature data according to the chronological order of occurrence of the scene feature data to obtain numerical embedded data.

[0116] For example, time dependency can be captured by using sequence models such as Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM) or Recurrent Neural Network (GRU). And the embedding layer is concatenated to map discrete features to continuous vectors in high-dimensional space, thereby obtaining numerical embedded data.

[0117] And, it should be noted that if Figure 3As shown in FIG, since the text data can be directly input into the large model, there is no need to process the text data. Figure 3 As shown, if data of at least two data types are included, encoders corresponding to multiple data types can be executed in parallel, thereby improving the efficiency of determining user data.

[0118] Therefore, based on multimodal data, scene collection and integration are carried out, encoders are defined for four types of basic data: image, audio, text, and digital, and input into the large model embedding space. This solves the problems of the existing technology that the source of user behavior data is not comprehensive, cannot meet the needs of cross-time and cross-region data integration, and cannot distinguish between daily behaviors and important behaviors.

[0119] In addition, based on the screened important behaviors, the basic data embedding matrix is ​​input into the big model, and the prompt words are used to let the big model select the content related to the behavioral data in the basic data, and return the basic data input embedding matrix index. Based on this, the above step 1302 can specifically include step 13021 and step 13024.

[0120] Step 13021, perform semantic analysis on the behavior description data through the big model to obtain the user behavior intention and context environment information represented by the behavior description data.

[0121] Exemplarily, the intent and context of the behavior data are analyzed through the prompt words in the large model, and the intent of the behavior data is identified through the prompt words to analyze the intention behind the user's behavior. For example, a user may take a photo at a specific time to record an important event. The context in which the behavior data occurs, including time, location, participants, event type, etc., is analyzed to enhance the accuracy of intent identification. The behavior data is indexed, including timestamps, behavior types, behavior duration, etc., for fast retrieval and matching.

[0122] Step 13022: extract user behavior element data associated with the user behavior intention from the user data of each data type associated with the behavior description data.

[0123] For example, image, audio, text and digital data encoders are combined to extract important cross-modal elements, such as key objects in images, emotional intonation in audio, and emotional vocabulary in text. An attention layer is added after the large model embedding matrix, and the attention mechanism is used to allow the large model to automatically allocate more attention resources to areas with higher correlation with behavioral data when processing the embedding matrix.

[0124] Step 13023, scoring the user behavior element data according to the context information to obtain a relevance score of the user behavior element.

[0125] Exemplarily, the relevance of the basic data is scored by the prompt words, that is, the relevance of the basic data and the behavioral data is scored by the big model through the context analysis results.

[0126] Step 13024, filtering the first multimodal behavior data from the user behavior element data, wherein the relevance score of the first multimodal behavior data is greater than or equal to a preset threshold.

[0127] Therefore, the data is automatically sorted and aggregated based on the big model, and the basic data related to the user behavior data is selected through the big model's understanding of multimodal data, so as to solve the problem that the existing technology lacks sentiment analysis and it is difficult for users to empathize with the aggregated results.

[0128] Regarding step 140 , in the embodiment of the present application, step 140 may specifically include step 1401 and step 1402 .

[0129] Step 1401: classify and summarize the first multimodal behavior data according to the behavior description data to obtain multimodal summarized behavior data.

[0130] Step 1402, associating the behavior description data with the multimodal summary behavior data to obtain user memory data.

[0131] Exemplarily, the behavior description data can be associated with the multimodal aggregated behavior data to obtain user memory data, and the user memory data can be stored and provided to various types of applications for scenario analysis and data display.

[0132] In summary, the method of automatically integrating multimodal behavior data into important moments of users in the embodiments of the present application is to understand them through automatic fusion. Through the powerful understanding ability of the big model, with the help of electronic devices, a variety of user data can be perceived every day, and can be automatically aggregated and understood, so as to automatically organize and build the user's important scenario library, provide users with a convenient emotional memory management solution, and bring users better value of self-growth.

[0133] It should be noted that the data processing method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, wearable devices, etc. In some embodiments of the present application, the data processing method provided in the embodiments of the present application is described by taking the electronic device as the execution subject to execute the data processing method as an example.

[0134] The data processing method provided in the embodiment of the present application can be executed by a data processing device. In the embodiment of the present application, a data processing device executing the data processing method is taken as an example to illustrate the device of the data processing method provided in the embodiment of the present application.

[0135] The present application also provides a data processing device. Figure 4 Provide detailed explanation.

[0136] Figure 4 A schematic diagram of the structure of a data processing device provided for some embodiments of the present application.

[0137] like Figure 4 As shown, the data processing device 40 can be applied to an electronic device, and the data processing device 40 can specifically include:

[0138] The acquisition module 401 is used to acquire multimodal behavior data.

[0139] The determination module 402 is used to determine the behavior description data of the user behavior according to the multimodal behavior data.

[0140] The screening module 403 is used to screen the first multimodal behavior data associated with the behavior description data from the multimodal behavior data.

[0141] The generating module 404 is used to generate user memory data corresponding to the user behavior based on the behavior description data and the first multimodal behavior data.

[0142] The data processing device 40 in the embodiment of the present application is described in detail below, as shown below.

[0143] In some embodiments of the present application, the determination module 402 is further used to determine the user behavior according to the multimodal behavior data, and the user behavior includes at least one of the following: a first user behavior corresponding to the user's experience time, a second user behavior corresponding to the user's experience location, and a third user behavior corresponding to the user's experience event; wherein the user's experience time includes at least one of the following: a commemorative day marked by the user, a holiday provided by the device system, and a public holiday provided by the device system; the experience location includes at least one of the following: a temporary location of the user, a commemorative location marked by the user; the user's experience event includes at least one of the following: a non-routine event, a planned event;

[0144] The generating module 404 is also used to generate behavior description data of the user behavior according to the user behavior.

[0145] In some embodiments of the present application, the determination module 402 is further used to determine the user's permanent location and the user's movement trajectory in each of the N time periods according to the location data of the user's location in the N time periods, when the multimodal behavior data includes location data of the user's location in N time periods, the location data includes location data, location type, and user residence time, and the second user behavior includes the user's behavior at the user's temporary location, and N is an integer greater than 1;

[0146] The screening module 403 is also used to screen out the temporary locations of the users from the user movement trajectories in each time period based on the permanent locations of the users;

[0147] The determination module 402 is further configured to determine a behavior of the user at the user's temporary location as a second user behavior.

[0148] In some embodiments of the present application, the data processing device 40 further includes a fitting module, which is used to fit the location data of the user's location in N time periods by a density clustering algorithm to obtain the user's stay location in each time period;

[0149] The generating module 404 is further used to generate the user movement trajectory of each time period based on the user's stay location in each time period;

[0150] The screening module 403 is also used to screen the user's permanent location from the user's stay locations, wherein the user's permanent location satisfies at least one of the following conditions: the location data of the user's permanent location appears in N user movement trajectories a number of times greater than or equal to a preset number of times, the location type of the user's permanent location matches the preset location type, and the user's stay time at the user's permanent location matches the preset stay time.

[0151] In some embodiments of the present application, the determination module 402 is specifically used to determine the user's daily behavior data by a frequent mining algorithm based on the multimodal behavior data when the third user behavior includes a behavior corresponding to a non-daily event, and the user's daily behavior data includes the user's regular or frequently occurring behavior data;

[0152] Based on the user's daily behavior data, determine the user's non-daily behavior data from the multimodal behavior data;

[0153] Analyze users' non-routine behavior data to obtain non-routine events;

[0154] The behavior corresponding to the non-routine event is determined as the third user behavior.

[0155] In some embodiments of the present application, the determination module 402 is specifically configured to, when the third user behavior includes a behavior corresponding to a planned event, filter data of a user's pre-planned event from the multimodal behavior data;

[0156] Perform data analysis on the data of the user's pre-planned events to obtain the planned events;

[0157] The behavior corresponding to the planned event is determined as the third user behavior.

[0158] In some embodiments of the present application, the screening module 403 is specifically used to, when the multimodal behavior data includes data of at least one of the following data types: image data, audio data, text data, and numerical data, process the data corresponding to each type in the multimodal behavior data through an encoder corresponding to each data type to obtain user data associated with the behavior description data under each type;

[0159] The user data associated with the behavior description data under each type is associated through the large model to obtain the first multimodal behavior data.

[0160] In some embodiments of the present application, the screening module 403 is specifically used to, when the encoder includes a visual data encoder and the user data includes image embedded data, match the aspect ratio of the image data with a preset aspect ratio through the visual data encoder to obtain a target preset aspect ratio matching the image data;

[0161] According to a target preset aspect ratio, performing image segmentation processing on an image corresponding to the image data to obtain at least two image segments;

[0162] Performing pixel shuffling processing on at least two image fragments to obtain at least two rearranged image fragments, wherein the resolution of the rearranged image fragments is greater than the resolution of the image;

[0163] The data dimensions of at least two rearranged image segments are adjusted to the preset input data dimensions of the large model to obtain image embedding data.

[0164] In some embodiments of the present application, the screening module 403 is specifically used to extract the user audio feature from the audio waveform of the audio data when the encoder includes an audio data encoder and the user data includes audio embedded data;

[0165] Through a small convolutional neural network, the user audio features are converted into audio conversion data, and the feature dimension of the audio conversion data is smaller than the feature dimension of the audio data;

[0166] The audio conversion data is discretized through a multi-scale residual vectorization algorithm to obtain audio embedding data.

[0167] In some embodiments of the present application, the screening module 403 is specifically used to, when the encoder includes a numerical data encoder and the user data includes numerical embedded data, adjust the numerical data according to preset standardized quantitative parameters through the numerical data encoder to obtain standardized quantitative numerical data;

[0168] According to the data type of the numerical data, scene feature data corresponding to the data type is extracted from the standardized quantitative numerical data;

[0169] According to the chronological order of the occurrence of the scene feature data, the scene feature data is reconstructed to obtain numerical embedded data.

[0170] In some embodiments of the present application, the screening module 403 is specifically used to perform semantic analysis on the behavior description data through a large model to obtain the user behavior intention and context information represented by the behavior description data;

[0171] Extracting user behavior element data associated with user behavior intention from user data of each data type associated with the behavior description data;

[0172] Score the user behavior element data according to the context information to obtain the relevance score of the user behavior element;

[0173] First multimodal behavior data is filtered from the user behavior element data, and a relevance score of the first multimodal behavior data is greater than or equal to a preset threshold.

[0174] In some embodiments of the present application, the generating module 404 is specifically used to classify and summarize the first multimodal behavior data according to the behavior description data to obtain multimodal summary behavior data;

[0175] The behavior description data is associated with the multimodal summary behavior data to obtain user memory data.

[0176] The data processing device 40 in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0177] The data processing device 40 in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0178] The data processing device 40 provided in the embodiment of the present application can realize Figures 1 to 3 The various processes implemented in the data processing method embodiment shown achieve the same technical effect, and will not be described again here to avoid repetition.

[0179] Based on this, the data processing device provided in the embodiment of the present application determines the behavior description data of the user behavior through the multimodal behavior data of the user, and selects the first multimodal behavior data associated with the behavior description data from the multimodal behavior data, and then generates the user memory data corresponding to the user behavior based on the behavior description data and the first multimodal behavior data. In this way, by collecting multimodal behavior data, the types of data elements can be enriched, the comprehensiveness and completeness of the data obtained from the user behavior can be improved, and the accuracy of determining the source of the user behavior data can be improved, and the first multimodal behavior data can be summarized and sorted according to the behavior description data, without the need for the user to manually edit the information of the user's life experience, thereby reducing the complexity of information editing and improving the efficiency of information editing.

[0180] Optional, such as Figure 5 As shown, an embodiment of the present application also provides an electronic device 50, including a processor 501 and a memory 502, wherein the memory 502 stores programs or instructions that can be executed on the processor 501, and when the program or instructions are executed by the processor 501, the various steps of the above-mentioned data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0181] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0182] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0183] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, a processor 610 and other components.

[0184] Those skilled in the art will appreciate that the electronic device 600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system.

[0185] Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0186] In the embodiment of the present application, the processor 610 is used to obtain multimodal behavior data. The processor 610 is also used to determine the behavior description data of the user behavior based on the multimodal behavior data. The processor 610 is also used to filter the first multimodal behavior data associated with the behavior description data from the multimodal behavior data. The processor 610 is also used to generate user memory data corresponding to the user behavior based on the behavior description data and the first multimodal behavior data.

[0187] The electronic device 600 is described in detail below, as shown below.

[0188] In some embodiments of the present application, in some embodiments of the present application, the processor 610 is further used to determine the user behavior according to the multimodal behavior data, and the user behavior includes at least one of the following: a first user behavior corresponding to the user's experience time, a second user behavior corresponding to the user's experience location, and a third user behavior corresponding to the user's experience event; wherein the user's experience time includes at least one of the following: a commemorative day marked by the user, a holiday provided by the device system, and a public holiday provided by the device system; the experience location includes at least one of the following: a temporary location of the user, a commemorative location marked by the user; the user's experience event includes at least one of the following: a non-routine event, a planned event;

[0189] Based on the user behavior, behavior description data of the user behavior is generated.

[0190] In some embodiments of the present application, the processor 610 is further configured to, when the multimodal behavior data includes location data of a location where the user locates within N time periods, the location data includes location data, location type, and user residence time, and the second user behavior includes the behavior of the user at the user's temporary location, and N is an integer greater than 1, determine the user's permanent location and the user's movement trajectory in each of the N time periods according to the location data of the location where the user locates within the N time periods;

[0191] Based on the user's permanent location, filter out the user's temporary location from the user's movement trajectory in each time period;

[0192] The behavior of the user at the user's temporary location is determined as the second user behavior.

[0193] In some embodiments of the present application, the processor 610 is further configured to fit the location of the user in each time period through a density clustering algorithm according to the location data of the location of the user in N time periods, so as to obtain the user's stay location in each time period;

[0194] Generate the user movement trajectory for each time period based on the user's stay location in each time period;

[0195] The user's permanent location is filtered from the user's stay locations, wherein the user's permanent location satisfies at least one of the following conditions: the number of times the location data of the user's permanent location appears in N user movement trajectories is greater than or equal to a preset number of times, the location type of the user's permanent location matches the preset location type, and the user's stay time at the user's permanent location matches the preset stay time.

[0196] In some embodiments of the present application, the processor 610 is further configured to, when the third user behavior includes a behavior corresponding to a non-daily event, determine the user's daily behavior data by a frequent mining algorithm based on the multimodal behavior data, where the user's daily behavior data includes the user's regular or frequently occurring behavior data;

[0197] Based on the user's daily behavior data, determine the user's non-daily behavior data from the multimodal behavior data;

[0198] Analyze users' non-routine behavior data to obtain non-routine events;

[0199] The behavior corresponding to the non-routine event is determined as the third user behavior.

[0200] In some embodiments of the present application, the processor 610 is further configured to, when the third user behavior includes a behavior corresponding to a planned event, filter data of a user's pre-planned event from the multimodal behavior data;

[0201] Perform data analysis on the data of the user's pre-planned events to obtain the planned events;

[0202] The behavior corresponding to the planned event is determined as the third user behavior.

[0203] In some embodiments of the present application, the processor 610 is further configured to, when the multimodal behavior data includes data of at least one of the following data types: image data, audio data, text data, and numerical data, process the data corresponding to each type in the multimodal behavior data through an encoder corresponding to each data type to obtain user data associated with the behavior description data under each type;

[0204] The user data associated with the behavior description data under each type is associated through the large model to obtain the first multimodal behavior data.

[0205] In some embodiments of the present application, the processor 610 is further configured to, when the encoder includes a visual data encoder and the user data includes image embedded data, match the aspect ratio of the image data with a preset aspect ratio through the visual data encoder to obtain a target preset aspect ratio matching the image data;

[0206] According to a target preset aspect ratio, performing image segmentation processing on an image corresponding to the image data to obtain at least two image segments;

[0207] Performing pixel shuffling processing on at least two image fragments to obtain at least two rearranged image fragments, wherein the resolution of the rearranged image fragments is greater than the resolution of the image;

[0208] The data dimensions of at least two rearranged image segments are adjusted to the preset input data dimensions of the large model to obtain image embedding data.

[0209] In some embodiments of the present application, the processor 610 is further configured to, when the encoder includes an audio data encoder and the user data includes audio embedded data, extract the user audio feature from the audio waveform of the audio data;

[0210] Through a small convolutional neural network, the user audio features are converted into audio conversion data, and the feature dimension of the audio conversion data is smaller than the feature dimension of the audio data;

[0211] The audio conversion data is discretized through a multi-scale residual vectorization algorithm to obtain audio embedding data.

[0212] In some embodiments of the present application, the processor 610 is further configured to, when the encoder includes a numerical data encoder and the user data includes numerical embedded data, adjust the numerical data according to a preset standardized quantitative parameter through the numerical data encoder to obtain standardized quantitative numerical data;

[0213] According to the data type of the numerical data, scene feature data corresponding to the data type is extracted from the standardized quantitative numerical data;

[0214] According to the chronological order of the scene feature data, the scene feature data is reconstructed to obtain numerical embedded data.

[0215] In some embodiments of the present application, the processor 610 is further configured to perform semantic analysis on the behavior description data through a large model to obtain user behavior intention and context information represented by the behavior description data;

[0216] Extracting user behavior element data associated with user behavior intention from user data of each data type associated with the behavior description data;

[0217] Score the user behavior element data according to the context information to obtain the relevance score of the user behavior element;

[0218] The first multimodal behavior data is filtered from the user behavior element data, and the relevance score of the first multimodal behavior data is greater than or equal to a preset threshold.

[0219] In some embodiments of the present application, the processor 610 is further configured to classify and aggregate the first multimodal behavior data according to the behavior description data to obtain multimodal aggregated behavior data;

[0220] The behavior description data is associated with the multimodal summary behavior data to obtain user memory data.

[0221] It should be understood that the input unit 604 may include a graphics processing unit (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of a static image or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 606 may include a display panel, and the display panel may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and at least one of other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch display. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as a volume display button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0222] The memory 609 can be used to store software programs and various data. The memory 609 can mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area can store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 609 can include a volatile memory or a non-volatile memory, or the memory 609 can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 609 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0223] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless display signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 610.

[0224] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0225] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0226] In addition, an embodiment of the present application further provides a chip, which includes a processor and a display interface, the display interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned data processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0227] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0228] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned data processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0229] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0230] In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0231] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0232] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A data processing method, characterized in that: include: Acquire multimodal behavioral data; Determining behavior description data of the user behavior based on the multimodal behavior data; Filtering first multimodal behavior data associated with the behavior description data from the multimodal behavior data; Based on the behavior description data and the first multimodal behavior data, user memory data corresponding to the user behavior is generated.

2. The method according to claim 1, characterized in that Determining behavior description data of the user behavior according to the multimodal behavior data includes: Determine the user behavior according to the multimodal behavior data, wherein the user behavior includes at least one of the following: a first user behavior corresponding to a user-experienced time, a second user behavior corresponding to a user-experienced location, and a third user behavior corresponding to a user-experienced event; wherein the user-experienced time includes at least one of the following: a commemorative day marked by the user, a holiday provided by the device system, and a public holiday provided by the device system; the experience location includes at least one of the following: a user's temporary location, a commemorative location marked by the user; the user-experienced event includes at least one of the following: a non-routine event, and a planned event; Based on the user behavior, behavior description data of the user behavior is generated.

3. The method according to claim 1, characterized in that The multimodal behavior data includes data of at least one of the following data types: image data, audio data, text data, and numerical data; The step of selecting the first multimodal behavior data associated with the behavior description data from the multimodal behavior data includes: Processing the data corresponding to each type in the multimodal behavior data through an encoder corresponding to each data type to obtain user data associated with the behavior description data under each type; The user data associated with the behavior description data under each type is associated through the large model to obtain the first multimodal behavior data.

4. The method according to claim 3, characterized in that The step of associating the user data associated with the behavior description data under each type through the large model to obtain the first multimodal behavior data includes: By using the large model, semantic analysis is performed on the behavior description data to obtain user behavior intention and context environment information represented by the behavior description data; Extracting user behavior element data associated with the user behavior intention from the user data of each data type associated with the behavior description data; Scoring the user behavior element data according to the context information to obtain a relevance score of the user behavior element; The first multimodal behavior data is filtered from the user behavior element data, and a relevance score of the first multimodal behavior data is greater than or equal to a preset threshold.

5. The method according to claim 1, characterized in that The generating user memory data corresponding to the user behavior based on the behavior description data and the first multimodal behavior data includes: According to the behavior description data, the first multimodal behavior data is classified and summarized to obtain multimodal summary behavior data; The behavior description data is associated with the multimodal summary behavior data to obtain the user memory data.

6. A data processing device, characterized in that: include: An acquisition module, used to acquire multimodal behavior data; A determination module, used to determine behavior description data of the user behavior based on the multimodal behavior data; A screening module, configured to screen first multimodal behavior data associated with the behavior description data from the multimodal behavior data; A generation module is used to generate user memory data corresponding to the user behavior based on the behavior description data and the first multimodal behavior data.

7. The device according to claim 6, characterized in that The determination module is further used to determine the user behavior according to the multimodal behavior data, wherein the user behavior includes at least one of the following: a first user behavior corresponding to a user-experienced time, a second user behavior corresponding to a user-experienced location, and a third user behavior corresponding to a user-experienced event; wherein the user-experienced time includes at least one of the following: a commemorative day marked by the user, a holiday provided by the device system, and a public holiday provided by the device system; the experience location includes at least one of the following: a temporary location of the user, a commemorative location marked by the user; and the user-experienced event includes at least one of the following: a non-routine event and a planned event; The generating module is further used to generate behavior description data of the user behavior according to the user behavior.

8. The device according to claim 6, characterized in that The screening module is specifically used for, when the multimodal behavior data includes at least one of the following data types: image data, audio data, text data, and numerical data, processing the data corresponding to each type in the multimodal behavior data through an encoder corresponding to each data type to obtain user data associated with the behavior description data under each type; The user data associated with the behavior description data under each type is associated through the large model to obtain the first multimodal behavior data.

9. The device according to claim 8, characterized in that The screening module is specifically used to perform semantic analysis on the behavior description data through the large model to obtain the user behavior intention and context environment information represented by the behavior description data; Extracting user behavior element data associated with the user behavior intention from the user data of each data type associated with the behavior description data; Scoring the user behavior element data according to the context information to obtain a relevance score of the user behavior element; The first multimodal behavior data is filtered from the user behavior element data, and a relevance score of the first multimodal behavior data is greater than or equal to a preset threshold.

10. The device according to claim 6, characterized in that The generating module is specifically used to classify and summarize the first multimodal behavior data according to the behavior description data to obtain multimodal summary behavior data; The behavior description data is associated with the multimodal summary behavior data to obtain the user memory data.

Images