Construction method and device of multi-terminal collaborative large model and electronic equipment

Abstract

The invention provides a construction method and device of a multi-terminal collaborative large model and electronic equipment, and relates to the field of data processing. The method comprises the following steps: acquiring to-be-processed data sent by a device group; according to the to-be-processed data, a model factor generator is adopted to generate a model factor unit; obtaining user behavior data corresponding to the user equipment from the to-be-processed data, wherein the equipment group comprises the user equipment; based on the user behavior data, performing variation on the model factor unit to obtain a multi-mode dynamic model factor; based on user equipment, migrating the multi-modal dynamic moduli to obtain a personalized moduli pool; and according to the personalized memetic pool, constructing a large multi-terminal collaborative model. By implementing the technical scheme provided by the invention, cooperative work of multiple devices is facilitated.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and specifically to a method, device and electronic device for constructing a multi-terminal collaborative large model. Background Art

[0002] As smart devices are becoming more and more popular, users are increasingly demanding multi-device collaboration, especially in scenarios such as smart homes, smart mobile terminals, and industrial IoT. For example, in smart homes, users want to use voice assistants to achieve unified control of devices such as TVs, air conditioners, and lights, while in mobile offices, users expect to be able to seamlessly switch tasks between mobile phones, tablets, and laptops.

[0003] At present, the existing multi-device collaboration technology mainly relies on general algorithms, that is, the traditional general model is designed with universality as the core and lacks the ability to dynamically adapt to personalized needs. For example, different users have significant differences in their usage habits and preferences for voice assistants, but it is difficult for existing models to quickly adjust the response logic according to the user's personalized behavior. Therefore, the general model built with general algorithms is not conducive to the collaboration of multiple devices.

[0004] Therefore, there is an urgent need for a method, device and electronic device for constructing a multi-terminal collaborative large model. Summary of the invention

[0005] The present application provides a method, device and electronic device for constructing a multi-terminal collaborative large model to facilitate collaborative work of multiple devices.

[0006] In a first aspect of the present application, a method for constructing a multi-terminal collaborative large model is provided, the method comprising: obtaining data to be processed sent by a device group; generating a meme unit using a meme generator based on the data to be processed; obtaining user behavior data corresponding to a user device from the data to be processed, the device group including the user device; based on the user behavior data, mutating the meme unit to obtain a multimodal dynamic meme; based on the user device, migrating the multimodal dynamic meme to obtain a personalized meme pool; and constructing a multi-terminal collaborative large model based on the personalized meme pool.

[0007] By adopting the above technical solution, meme units are optimized in real time based on user behavior data, and the logical structure of memes is adjusted through the mutation mechanism, so that they can quickly adapt to changes in user needs. This mechanism overcomes the limitations of the fixed logic of traditional models and can provide more personalized services. Each user device generates a personalized meme pool through user behavior data, retaining the knowledge of user-specific preferences. This personalized meme pool approach not only improves the model's ability to adapt to user needs, but also reduces the problem of over-reliance on global models. The meme generator can fuse data from different modalities to generate multimodal dynamic memes for processing complex cross-modal tasks. Meme units can be migrated between device groups, allowing the model to adapt to the characteristics of different devices and achieve seamless collaboration across devices. Meme units are lightweight knowledge expressions for specific tasks, with lower computational overhead than traditional large models, and are suitable for running on edge devices with limited resources. This design improves processing efficiency while reducing the requirements for device hardware performance, allowing low-power devices to run efficiently. The personalized meme pool and multi-terminal collaboration mechanism reduce dependence on central servers, and data processing is more completed on local devices, reducing data transmission delays and bandwidth requirements. Through the design of meme generation and personalized meme pool, most of the processing and optimization of user behavior data can be completed on local devices, reducing the frequency of data upload. This method improves the privacy protection of user data and reduces the risk of sensitive data leakage. In summary, it facilitates the collaborative work of multiple devices.

[0008] Optionally, the acquisition of the data to be processed sent by the device group specifically includes: acquiring visual data sent by an image acquisition device; acquiring voice data sent by a voice acquisition device; acquiring the user behavior data sent by a user device; performing data processing on the visual data, the voice data and the user behavior data to obtain the data to be processed, and the data processing includes denoising, filtering and normalization.

[0009] By adopting the above technical solutions, denoising can effectively filter out random noise and irrelevant interference in the data, such as ambient light interference in image data and background noise in voice data, so as to ensure the authenticity and reliability of the data. The quality of the model input data is improved, providing more accurate basic information for the generation and variation of subsequent meme units. Filtering can further smooth the data and eliminate high-frequency interference or abnormal fluctuations. For example, small jitters in images or high-frequency noise in voice signals can be eliminated by low-pass filtering. In the processing of user behavior data, filtering can effectively remove the jitter of sensor signals in motion capture and improve the accuracy of behavior recognition. Normalization can map different data modes to a unified scale, thereby improving the consistency and comparability of data. This not only helps the fusion of multimodal data, but also accelerates the convergence speed of subsequent algorithms and optimizes model performance. The visual data provided by the image acquisition device can contain rich environmental context and user status information. By processing the visual data, clear environmental perception capabilities can be provided for model construction. The voice data provided by the voice acquisition device can capture user intentions. Denoising and normalization processing can improve the accuracy of voice recognition, enable it to better cooperate with visual data, and support multimodal tasks. The processed behavioral data can be combined with visual and voice data to form multimodal dynamic memes. Integrating visual, voice and user behavior data into unified data to be processed can make up for the lack of information in single modality data and improve the model's ability to understand user behavior and intentions.

[0010] Optionally, generating a meme unit using a meme generator according to the data to be processed specifically includes: mapping data features corresponding to the data to be processed to a pre-trained general meme pool through the meme generator, determining a task type, the task types including image classification, speech recognition, and user behavior recognition; based on the task type, activating the meme unit.

[0011] By adopting the above technical solution, the features of the data to be processed are mapped to the pre-trained universal meme pool, which means that each type of data can be mapped to a unified representation space through mapping. This mapping provides the same basic framework for different modal data, reduces the differences between modalities, and makes cross-modal collaboration more efficient. Through data feature mapping, the meme generator can automatically extract valuable features from a variety of input data, eliminating the need for manual feature selection or complex feature engineering, thereby improving processing efficiency. This automated process provides a simpler path for the integration and analysis of multimodal data, especially for complex data scenarios. By automatically determining the task type, it can be ensured that the meme generator selects the most appropriate processing method based on the specific task objectives. Automatic identification of task types means that the corresponding meme units can be specifically selected and optimized for each task. In this way, the system can efficiently and accurately process different tasks, avoiding information loss or error processing during task processing. Activating the corresponding meme units based on the task type provides great flexibility. Different tasks correspond to different meme units, ensuring the customization of the processing flow. This activation mechanism can automatically load and start the most suitable model components according to the needs of specific tasks, improving the personalization and efficiency of task processing. Through task type identification and meme unit activation, the system can customize the optimized path for each task without adding too much computing burden. This can reduce unnecessary waste of computing resources and thus improve the overall processing speed.

[0012] Optionally, based on the user behavior data, the meme unit is mutated to obtain a multimodal dynamic meme, which specifically includes: determining an initial meme logic based on the meme unit; mutating the initial meme logic according to the user behavior data to obtain a target meme logic; sending the target meme logic to a target device, and obtaining output data sent by the target device and generated by the target device after processing according to the target meme logic; comparing the output data with actual data to obtain a comparison result.

[0013] By adopting the above technical solution, by mutating the meme unit based on user behavior data, the system can capture and adapt to the unique behavior patterns of each user. By comparing the output data of the target device, the system can evaluate the accuracy of the target meme logic, thereby continuously iterating and optimizing the meme logic. This closed-loop feedback mechanism enables the model to optimize its prediction and task execution capabilities based on the actual operation results. The system continuously adjusts and improves the meme logic based on the comparison results to ensure that the meme logic finally generated is more in line with the actual needs of the target device, thereby improving the accuracy and execution efficiency of the overall system. Through the generation of multimodal dynamic memes, the system is able to process inputs from different devices and data sources. In this passage, by mutating the meme unit, the system can adjust the processing method in real time to adapt to the changes in multimodal data. Different devices and environments may have different needs and processing methods, and dynamic memes can be adjusted according to the characteristics of the target device. The mutation process of the meme unit enables the system to adapt to a variety of devices and tasks, and improves the ability of multiple devices to work together. By sending the target meme logic to the target device and obtaining its processed output data, the system can obtain feedback from the target device in real time. This process provides a direct verification basis for the variation of meme units, thus avoiding the deviation between theory and actual operation. The real-time feedback mechanism enhances the system's control over the task execution process, enabling the system to self-adjust in actual operation, reduce errors, and improve the accuracy of task execution.

[0014] Optionally, based on the user behavior data, the meme unit is mutated to obtain a multimodal dynamic meme, which specifically also includes: using a meme fitness function to calculate a feedback data group, the feedback data group includes a task completion rate, user click data and delay data; by adjusting the fitness corresponding to the meme fitness function, the feedback data group is logically corrected to obtain a correction result; if it is determined that the comparison result indicates that the output data is consistent with the actual data, and the correction result is normal, then the meme unit is assigned a weight through an attention mechanism to generate the multimodal dynamic meme.

[0015] By adopting the above technical solution, by introducing feedback data such as task completion rate, user click data and delay data, the system can comprehensively evaluate the effect of user operation and system response. These data provide a basis for the variation and optimization of meme units, thereby ensuring that the system is constantly adjusted according to the actual operation results. This feedback mechanism enables the system to accurately identify which operations are helpful to users and which parts may have problems, thereby helping the system to self-adjust and improve the accuracy of task execution. The meme fitness function is used to evaluate the fitness of the meme unit, and is adjusted in combination with the feedback data to ensure that the meme unit generated after each iteration can better meet the actual task requirements. This adaptive optimization mechanism enhances the intelligence of the system, enabling it to make real-time adjustments based on user behavior data, thereby improving the user experience. By calculating the fitness and making corrections, the system can minimize the impact of inappropriate meme units and ensure that the generated memes are more targeted. Feedback data can help the system identify bottlenecks in tasks, thereby optimizing meme logic and improving the overall completion rate of task execution. User click data reflects user interests and operation paths, while delay data reveals the efficiency of system response. These data help optimize the system's reaction speed and interaction design, ensuring that the system can respond more quickly and accurately according to user behavior. By adjusting the meme fitness function, the system can make logical corrections to the feedback data and obtain the correction results. This correction mechanism ensures that the system can adjust according to the difference between the actual data and the expected results during execution, minimizing the deviation. The system can dynamically adjust the meme unit during the execution of the task to ensure that key indicators such as task completion rate and latency remain in the best state. The system's adaptive ability can be optimized in real time according to the changing user behavior and task environment, improving its fault tolerance and responsiveness to environmental changes. Once it is confirmed that the output data is consistent with the actual data and the meme correction is normal, the system will assign appropriate weights to the meme unit through the attention mechanism. The attention mechanism enables the system to adjust the influence of the meme unit according to the priority and importance of the task, so as to focus on key tasks and optimize resource allocation.

[0016] Optionally, based on the user device, the multimodal dynamic meme is migrated to obtain a personalized meme pool, specifically including: obtaining the resource status corresponding to the user device; if it is determined that the resource status corresponding to the user device does not meet the usage conditions, obtaining an associated device of the user device; sending the multimodal dynamic meme to the associated device; and determining the personalized meme pool based on the meme unit corresponding to the user device and the multimodal dynamic meme corresponding to the associated device.

[0017] By adopting the above technical solution, the system first evaluates the resource status of the user's device to ensure that the device can effectively perform tasks within its resource limits. If the device resources are insufficient, the system will automatically migrate the meme to other devices with more sufficient resources, thus avoiding the problem of task failure or slow execution due to resource limitations. This adjustment based on resource status can avoid device overload and resource waste, and improve the efficiency of system operation, especially in scenarios where multiple devices work together. If the user's device resources cannot meet the operating requirements, the system will share the computing tasks by acquiring associated devices. This way of dynamically migrating meme units ensures that the system can flexibly allocate resources in a multi-device environment, improving the flexibility and resource utilization of the system. The migration process can allocate tasks to devices with more resources without affecting the quality of task completion, reduce the pressure on a single device, and improve the efficiency of task processing. By migrating memes based on the resource status of the user's device, the system can personalize the device status of each user. For example, a device with abundant resources can undertake more complex tasks, while a device with fewer resources can focus on lightweight tasks. This personalized resource allocation based on device capabilities ensures that each user can obtain the service that best suits their device configuration, thereby optimizing the overall user experience. The system creates a personalized meme pool by integrating the meme units of user devices and associated devices, combined with the actual status of device resources. The memes in the meme pool can be optimized for specific users and device environments, ensuring that users can get tailored responses in a multi-terminal collaborative environment. This personalized meme pool can help the system dynamically adjust according to the unique needs of each user, improve the accuracy of services and the efficiency of task execution. The system can share tasks through associated devices to ensure balanced workloads across the entire device group and improve the fluency and efficiency of multi-device collaboration.

[0018] Optionally, the method further includes: obtaining usage data of users when using smart home devices; inputting the usage data into the multi-terminal collaborative model to obtain a target meme group; and collaboratively controlling the smart home devices according to the target meme group.

[0019] By adopting the above technical solutions, the system can deeply understand the user's behavior and needs by obtaining the user's usage data in smart home in real time. This data-driven approach enables the device to adapt to the user's actual needs more intelligently. By analyzing these data, the system can automatically identify the user's operating habits, adjust the device configuration in real time, realize personalized and customized control, and provide services that are more in line with user needs. In a multi-device environment, traditional device control is often independent and cannot fully utilize the synergy between devices. Collaborative control of smart home devices through target meme groups can achieve seamless coordination between different devices. This collaborative control not only improves the coordination efficiency between devices, but also provides users with a more efficient and convenient smart home experience. By inputting user usage data into the multi-terminal collaborative large model, the system can generate a target meme group. This process enables the model to be dynamically adjusted according to actual usage and optimize the control strategy. The control logic of each device can be updated according to the user's historical behavior and real-time data, making the device respond faster and more accurately. Through the generation of the target meme group, the system can capture the user's behavior patterns in different scenarios, and continuously optimize the control strategy in a data-driven manner to improve the adaptability and intelligence level of the system. The system can continuously fine-tune the multi-terminal collaborative model based on the constantly updated user usage data. Each interaction between the user and the device provides new data for the model, thereby achieving continuous optimization and learning of the model. This learning ability ensures that the system can always adapt to the changing needs of users without falling into a fixed control logic.

[0020] In a second aspect of the present application, a device for constructing a multi-terminal collaborative large model is provided, the construction device comprising an acquisition module and a processing module, wherein the acquisition module is used to acquire data to be processed sent by a device group; the processing module is used to generate a meme unit using a meme generator based on the data to be processed; the acquisition module is also used to acquire user behavior data corresponding to a user device from the data to be processed, and the device group includes the user device; the processing module is also used to mutate the meme unit based on the user behavior data to obtain a multimodal dynamic meme; the processing module is also used to migrate the multimodal dynamic meme based on the user device to obtain a personalized meme pool; the processing module is also used to construct the multi-terminal collaborative large model based on the personalized meme pool.

[0021] In the third aspect of the present application, an electronic device is provided, which includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs the method described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, the method described above is executed.

[0023] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages: Meme units are optimized in real time based on user behavior data, and the logical structure of memes is adjusted through mutation mechanisms, so that they can quickly adapt to changes in user needs. This mechanism overcomes the limitations of the fixed logic of traditional models and can provide more personalized services. Each user device generates a personalized meme pool through user behavior data, retaining the knowledge of user-specific preferences. This personalized meme pool approach not only improves the model's ability to adapt to user needs, but also reduces the problem of over-reliance on global models. The meme generator can fuse data from different modalities to generate multimodal dynamic memes for processing complex cross-modal tasks. Meme units can be migrated between device groups, allowing models to adapt to the characteristics of different devices and achieve seamless collaboration across devices. Meme units are lightweight knowledge expressions for specific tasks, with lower computational overhead than traditional large models, and are suitable for running on edge devices with limited resources. This design reduces the requirements for device hardware performance while improving processing efficiency, allowing low-power devices to run efficiently. The personalized meme pool and multi-terminal collaboration mechanism reduce dependence on central servers, and data processing is more completed on local devices, reducing data transmission delays and bandwidth requirements. Through the design of meme generation and personalized meme pool, most of the processing and optimization of user behavior data can be completed on local devices, reducing the frequency of data upload. This method improves the privacy protection of user data and reduces the risk of sensitive data leakage. In summary, it facilitates the collaborative work of multiple devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a method for constructing a multi-terminal collaborative large model provided in an embodiment of the present application; Figure 2 Another flowchart of a method for constructing a multi-terminal collaborative large model provided in an embodiment of the present application; Figure 3 A schematic diagram of a module of a device for constructing a multi-terminal collaborative large model provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0025] Explanation of the reference numerals: 31, acquisition module; 32, processing module; 41, processor; 42, communication bus; 43, user interface; 44, network interface; 45, memory. DETAILED DESCRIPTION

[0026] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0027] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0028] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0029] With the rapid popularization of smart devices, users' demand for multi-device collaboration is increasing, especially in application scenarios such as smart homes, smart mobile terminals and industrial IoT. For example, in smart homes, users hope to use voice assistants to uniformly control TVs, air conditioners, lights and other devices; in mobile office scenarios, users expect to be able to seamlessly switch tasks between mobile phones, tablets and laptops.

[0030] However, most of the current multi-device collaboration technologies rely on general algorithms. These traditional general models are mainly designed for wide applicability and lack the ability to quickly adapt to personalized needs. For example, different users have significant differences in their usage habits and preferences for voice assistants, and existing models find it difficult to dynamically adjust the response logic based on the behavioral characteristics of each user. Therefore, models built based on general algorithms often find it difficult to effectively support collaboration between multiple devices in practical applications.

[0031] In order to solve the above technical problems, the present application embodiment provides a method for constructing a multi-terminal collaborative large model, referring to Figure 1 , Figure 1A flowchart of a method for constructing a multi-terminal collaborative large model provided in an embodiment of the present application. The construction method is applied to a server and includes steps S110 to S160, which are as follows: S110: Obtain the data to be processed sent by the device group.

[0032] Specifically, a server is a computer system that centrally manages, processes, and stores data. Its main task is to receive and process data from different devices or clients. A device group refers to multiple devices working together in a network, and these devices can usually be connected through different technologies, such as Wi-Fi, Bluetooth, ZigBee, etc. In a device group, each device has its own functions and data collection capabilities. Processed data refers to data obtained from a device group that is not directly used for the final decision or output, but needs to go through a certain processing process. For example, the data may need to be cleaned, analyzed, transformed, or combined with other data to produce meaningful results.

[0033] For example, suppose there is a smart home system, which includes smart bulbs, air conditioners, temperature and humidity sensors, and voice assistants. In the embodiment of the present application, the server obtains the data to be processed through the following steps: the devices such as smart bulbs and air conditioners continuously send their working status and currently set data to the server. For example, the data sent by the smart bulb shows its current brightness and color temperature, while the air conditioner sends the current temperature, humidity and wind speed. The temperature and humidity sensor monitors the temperature and humidity in the environment, and it sends these real-time collected data, such as "the current temperature is 22°C and the humidity is 45%", to the server. The voice assistant device converts the user's voice command into text data and sends it to the server. For example, the user says "lower the air conditioner temperature by 2 degrees", and the voice assistant converts this command into data and transmits it. After the server receives the above data from each device, it analyzes these data as "data to be processed". For example, the server may need to adjust the settings of the air conditioner according to the user's instructions, and at the same time combine the sensor data to determine whether the lighting and temperature and humidity need to be adjusted, and finally make a unified control decision based on these processed data. In this example, the data to be processed obtained by the server from the device group includes status information from different devices, data input by users, and data provided by environmental sensors. All of these data need to be further processed to achieve collaborative work of multiple devices.

[0034] In a possible implementation, obtaining the data to be processed sent by the device group specifically includes: obtaining visual data sent by the image acquisition device; obtaining voice data sent by the voice acquisition device; obtaining user behavior data sent by the user device; performing data processing on the visual data, voice data and user behavior data to obtain the data to be processed, and the data processing includes denoising, filtering and normalization.

[0035] Specifically, image acquisition devices refer to devices with visual perception capabilities, such as cameras, smart doorbells, surveillance cameras, etc. Visual data is usually video frames, pictures, or features extracted from images. Visual data is usually used to analyze scenes, identify objects, or detect actions. Voice acquisition devices refer to devices that can record voice information, such as microphones, voice assistants, smart speakers, etc. Voice data is usually user voice commands or environmental sounds. Voice data is used to identify user intentions, analyze voice content, or perform voice interaction tasks. User devices refer to devices that are directly operated by users, such as smartphones, tablets, remote controls, etc. User behavior data includes click operations, touch screen trajectories, device usage records, etc. These data can reflect user behavior patterns and preferences and provide a basis for personalized services. Remove useless information or interference from the data, such as removing noise from images and removing environmental noise from voice. Smoothing data can eliminate outliers or mutations in a short period of time, such as edge-preserving filtering of images and low-pass filtering of audio. Adjust the data to a unified range or scale to ensure the comparability of data from different devices, such as normalizing pixel values ​​to the [0, 1] interval or standardizing user behavior data.

[0036] S120: Generate a meme unit using a meme generator according to the data to be processed.

[0037] Specifically, the meme generator is an intelligent algorithm tool that can generate meme units related to specific tasks based on the characteristics of the input data and combined with a pre-trained knowledge base or model. The core of the meme generator is to match the input data characteristics and task requirements, and generate suitable logic units through analysis and reasoning. The meme unit is the task logic extracted from the meme generator and is used to handle a specific type of problem. It can be a rule, an algorithm module, or a trained model fragment. For example, for "voice data", the meme unit may be the "speech to text" logic; for "visual data", it may be the "object detection" algorithm. The generation of meme units is to decompose complex tasks and convert information from different data sources into targeted operation modules. This approach improves the scalability and efficiency of the system.

[0038] In a possible implementation, a meme generator is used to generate a meme unit according to the data to be processed, specifically including: mapping the data features corresponding to the data to be processed to a pre-trained general meme pool through the meme generator, determining the task type, which includes image classification, speech recognition, and user behavior recognition; based on the task type, activating the meme unit.

[0039] Specifically, the main task of the meme generator is to determine the task type to which the data belongs by analyzing the characteristics of the data to be processed, and extract the corresponding meme unit from the pre-trained general meme pool. The meme generator extracts the characteristics of the data to be processed, such as the texture and color distribution of image data, the spectral characteristics of voice data, and the operation mode of user behavior data. Based on these characteristics, the server determines the type of task that the data needs to perform, such as image data needs to be classified, voice data needs to be recognized, and user behavior data needs to be predicted. The meme pool stores general logic or models for various tasks, and through task type matching, the relevant meme units are activated and loaded into the current task environment. The general meme pool is a pre-built knowledge base that contains multiple task-related logic units or model fragments. Each meme unit is optimized for a specific task, such as a visual recognition module, a speech processing module, etc. The server uses the meme generator to determine the type of task that the current data needs to complete. Common types include: Image classification: identifying objects or scenes in an image, such as classifying "fruit" as "apple", "banana", etc.; Voice recognition: converting voice data into text or instructions, such as "play music" or "turn on the light"; User behavior recognition: analyzing user operation behavior and predicting their next operation, such as the sequence of buttons clicked by the user. Activation refers to selecting and loading the corresponding logic unit from the meme pool so that it starts processing data related to the current task.

[0040] S130: Obtain user behavior data corresponding to the user device from the data to be processed, where the device group includes the user device.

[0041] Specifically, the user behavior data corresponding to the user device is the part of the data to be processed that is directly related to the user's operation or habits. For example, the operation record of the user clicking or sliding the screen; the content and pattern of the user's voice command; the time period or frequency of the user's use of the device. The server will filter the behavior data directly related to the user device, such as filtering out the control commands sent by the user through the mobile phone, or the user's voice instructions to the smart speaker.

[0042] S140. Based on the user behavior data, the meme unit is mutated to obtain a multimodal dynamic meme.

[0043] Specifically, meme mutation refers to the dynamic adjustment of the logic, parameters or structure of the meme unit to make it more suitable for the current user's behavior and needs. The server identifies the user's special needs or preferences based on the behavior data. Modify the task logic of the meme unit so that it can more accurately meet the user's needs. The mutated meme unit becomes a new dynamic meme, which is suitable for the user's current scenario. Dynamic memes are not only for a single data type, but can also be combined with multimodal data for collaborative processing.

[0044] In a possible implementation, a meme unit is mutated based on user behavior data to obtain a multimodal dynamic meme, which specifically includes: determining an initial meme logic based on the meme unit; mutating the initial meme logic according to the user behavior data to obtain a target meme logic; sending the target meme logic to a target device, and obtaining output data sent by the target device and generated by the target device after processing according to the target meme logic; comparing the output data with the actual data to obtain a comparison result.

[0045] Specifically, the initial meme logic is the predefined rules or model of the meme unit, and the operation is performed in a general way when processing tasks. For example, the standard parsing method of voice commands or the default classification algorithm in image recognition. The initial meme logic is adjusted or mutated using user behavior data. The rules, parameters or structure of the meme unit are modified according to the user's specific behavior pattern. For example, the parsing method of the voice assistant is optimized to make it more in line with the user's language habits. The mutated logic is the meme unit logic customized for the current user or task scenario. Apply the mutated meme logic to the target device, observe the processing results of the device under the new logic, and record its output data. The output data is the result generated by the target device under the action of the target meme logic. The actual data is the result expected by the user or the correct result in the real scenario. Finally, the server checks whether the two are consistent. If they are consistent, it means that the mutation of the target meme logic is valid; otherwise, further adjustment is required.

[0046] For example, suppose the meme unit built into the smart speaker has the following initial logic. When the user says "dim the lights", the speaker will reduce the brightness of the lights to 50%. The user is accustomed to adjusting the brightness of the lights to 30% and switching to warm yellow when watching movies at night. The user has expressed his preference for this brightness and color combination through multiple voice commands. The server adjusts the initial meme logic based on the user's behavior data and parses the default "dim the lights" command as "adjust the brightness to 30% and switch to warm yellow". The server sends the target meme logic to the smart lighting system, and after execution, the lighting system outputs the following results: the light brightness is adjusted to 30%; the light color is switched to warm yellow. The server compares the output data of the lighting system (brightness 30%, warm yellow) with the user's actual demand data (brightness 30%, warm yellow). The results are consistent, indicating that the target meme logic is effective.

[0047] In a possible implementation, based on user behavior data, a meme unit is mutated to obtain a multimodal dynamic meme, which specifically includes: using a meme fitness function to calculate a feedback data group, the feedback data group includes a task completion rate, user click data, and delay data; by adjusting the fitness corresponding to the meme fitness function, the feedback data group is logically corrected to obtain a correction result; if it is determined that the comparison result indicates that the output data is consistent with the actual data, and the correction result is normal, the meme unit is assigned weights through an attention mechanism to generate a multimodal dynamic meme.

[0048] Specifically, the meme fitness function is an evaluation function used to measure the effect of the meme unit in performing a specific task. The feedback data set consists of multiple indicators, such as the task completion rate, that is, the success rate of the meme unit in the task. User interaction data is used to reflect the user's satisfaction with the system behavior. Latency data, that is, the time it takes for the meme unit to respond in the task, measures its efficiency. These indicators are comprehensively evaluated through the fitness function to generate a feedback data set. If the meme unit performs poorly on some indicators, for example, the delay is too long or the task completion rate is low, the meme unit logic needs to be corrected. The direction of the correction is guided by the results of the fitness function to improve the performance of the meme unit and the user experience. Check whether the meme unit output data is consistent with the actual data to ensure the correctness of the mutation logic. Ensure that the corrected meme unit logic can run effectively in the actual task. Use the attention mechanism to assign weights to the meme units to highlight the meme units that perform well in the task. The weight reflects the importance of the meme unit in the multimodal task, and finally generates an optimized multimodal dynamic meme.

[0049] S150. Based on the user device, migrate the multimodal dynamic memes to obtain a personalized meme pool.

[0050] Specifically, the server needs to obtain relevant information about the user's device, including the device's hardware resources, such as CPU, memory, storage, network connection status, and the interactive relationship between the device and other associated devices. This information will determine the target and method of meme migration. Multimodal dynamic memes are meme units that have been adjusted based on user behavior data and contain multiple modalities, such as voice, vision, and behavioral task logic. The core purpose of migration is to adjust memes from a general form to a personalized form to more efficiently adapt to the specific usage scenarios of user devices. If the user's device itself has insufficient resources, the meme can be migrated to associated devices for collaborative execution. The migrated meme unit will be stored in a device-specific meme pool, forming a personalized resource collection that can be quickly called and supports multi-task collaboration. The personalized meme pool is the result of collaboration between user devices and associated devices, and is an efficient solution for specific user needs.

[0051] In a possible implementation, based on the user device, multimodal dynamic memes are migrated to obtain a personalized meme pool, which specifically includes: obtaining the resource status corresponding to the user device; if it is determined that the resource status corresponding to the user device does not meet the usage conditions, obtaining the associated device of the user device; sending the multimodal dynamic memes to the associated device; and determining the personalized meme pool based on the meme unit corresponding to the user device and the multimodal dynamic memes corresponding to the associated device.

[0052] Specifically, the resource status of the user device refers to the hardware and software capabilities currently available to the device, such as processor performance, memory usage, storage space, network bandwidth, etc. The server needs to evaluate these parameters in real time to determine whether the device can independently run multimodal dynamic memes. If the evaluation result shows that the resources of the user device are insufficient to meet the operating conditions, for example, insufficient memory, insufficient computing power, or bandwidth limitations, the server will look for other devices associated with the user device. Associated devices refer to other devices that are in the same network as the user device and can participate in task collaboration. Migrate multimodal dynamic memes to associated devices with sufficient resources to share the task processing load. During the migration process, it is necessary to ensure that the meme logic is compatible with the operating environment of the associated device, such as the device's operating system and computing power. The meme unit on the user device is combined with the dynamic meme processed in the associated device to form a complete personalized meme pool. This meme pool has the ability to work together and can dynamically call resources between devices according to specific task requirements.

[0053] For example, a user is using a smartwatch, mobile phone, and tablet device to collaboratively complete office tasks. The user wants to quickly start the meeting mode through voice commands, which involves multiple tasks: 1. The smartwatch is responsible for reminding the meeting time; 2. The mobile phone is responsible for starting the meeting application and sharing the screen; 3. The tablet device is used as the main meeting interface. The smartwatch has limited resources and insufficient memory to run complex meme logic. The mobile phone has medium resources and is suitable for processing voice and task scheduling. The tablet device has sufficient resources and its large screen and high computing power are suitable for processing complex meme logic. The server detects that the smartwatch cannot process the multimodal dynamic meme of "meeting mode" alone. According to the status of the associated devices, the memes related to task scheduling are migrated to the mobile phone. The memes related to meeting interface settings and data synchronization are migrated to the tablet device. Smartwatch meme pool: reminder task meme unit. Mobile phone meme pool: task scheduling and voice interaction meme unit. Tablet meme pool: meeting interface control and data synchronization meme unit. The user receives a meeting reminder on the smartwatch. The user issues a voice command "start meeting mode" through the mobile phone. The mobile phone calls the task scheduling meme, cooperates with the tablet device, and starts the conference interface. The tablet device handles complex tasks, presents the conference interface, and receives the screen sharing data sent by the mobile phone.

[0054] S160. Based on the personalized meme pool, a multi-terminal collaborative large model is constructed.

[0055] Specifically, the personalized meme pool is generated by processing and mutating the user behavior data in the previous step, and it contains meme units that adapt to user habits and needs. These meme units can flexibly adjust and optimize the behavior logic according to the user's device and task requirements. The server uses the information obtained from the personalized meme pool to build a large model. The large model is not a model running on a single device, but an intelligent model for cross-device collaboration. The purpose of this large model is to enable multiple devices to work together, rather than simply individual devices performing tasks alone. Each device, such as a mobile phone, a tablet, and a smart home device, plays a different role in the large model according to its own resources and task requirements. Multi-end collaboration refers to the collaboration between multiple devices, such as mobile phones, tablets, and TVs, to ensure that they can complete the user's tasks together. For example, in a smart home scenario, a mobile phone can control the lights, air conditioners, and other devices in the home, while a voice assistant can be used to control these devices through voice commands. In an embodiment of the present application, a multi-end collaborative large model means that the server can intelligently allocate and schedule tasks between devices according to the personalized needs of the user's device to ensure seamless collaboration between multiple devices.

[0056] In one possible implementation, refer to Figure 2 , Figure 2 Another flowchart of a method for constructing a multi-terminal collaborative model provided in an embodiment of the present application includes steps S210 to S230, which are as follows: S210, obtaining usage data of users when using smart home devices; S220, inputting the usage data into the multi-terminal collaborative model to obtain a target meme group; S230, collaboratively controlling the smart home devices according to the target meme group.

[0057] Specifically, the server first needs to collect data from smart home devices, especially the behavioral data of users interacting with these devices. For example, the user's operating habits, the response time of the device, and the user's control method of the device, such as through voice assistants, mobile phones, remote controls, etc. These data reflect the user's preferences and habits in specific environments and scenarios. The server inputs the collected user usage data into the trained multi-terminal collaborative big model. The big model may already contain information such as the collaboration logic between multiple devices, user behavior patterns, and the capabilities and performance of different devices. By processing this data, the big model can understand the user's behavior, identify the user's needs, and optimize the collaboration between devices. In this process, the big model can also be dynamically adjusted according to the user's specific operations and device conditions. After processing, the multi-terminal collaborative big model will output a "target meme group", that is, a set of optimized control logic or parameters. These control logics correspond to the best match between user needs and device capabilities. The content of the target meme group can include how to adjust device settings, such as volume, brightness, temperature, etc., when to start or stop certain functions, and even how to coordinate the work between them according to the status of the device. Finally, the server will generate specific control instructions based on the target meme group and send corresponding instructions to each smart home device to ensure that these devices can work according to the user's needs and the best coordination between devices. For example, when the user issues a "home theater mode" command, smart speakers, TVs, lights and other devices will adjust their respective states according to the instructions of the target meme group, work together and provide a seamless user experience.

[0058] This application also provides a device for constructing a multi-terminal collaborative large model, referring to Figure 3 , Figure 3 A module schematic diagram of a construction device for a multi-terminal collaborative large model provided in an embodiment of the present application. The construction device is a server, and the server includes an acquisition module 31 and a processing module 32, wherein the acquisition module 31 acquires the data to be processed sent by the device group; the processing module 32 uses a meme generator to generate a meme unit according to the data to be processed; the acquisition module 31 acquires user behavior data corresponding to the user device from the data to be processed, and the device group includes the user device; the processing module 32 mutates the meme unit based on the user behavior data to obtain a multimodal dynamic meme; the processing module 32 migrates the multimodal dynamic meme based on the user device to obtain a personalized meme pool; the processing module 32 constructs a multi-terminal collaborative large model based on the personalized meme pool.

[0059] In a possible implementation, the acquisition module 31 acquires the data to be processed sent by the device group, specifically including: the acquisition module 31 acquires the visual data sent by the image acquisition device; the acquisition module 31 acquires the voice data sent by the voice acquisition device; the acquisition module 31 acquires the user behavior data sent by the user device; the processing module 32 processes the visual data, voice data and user behavior data to obtain the data to be processed, and the data processing includes denoising, filtering and normalization.

[0060] In a possible implementation, the processing module 32 generates a meme unit using a meme generator according to the data to be processed, specifically including: the processing module 32 maps the data features corresponding to the data to be processed to a pre-trained general meme pool through the meme generator, and determines the task type, which includes image classification, speech recognition, and user behavior recognition; the processing module 32 activates the meme unit based on the task type.

[0061] In a possible implementation, the processing module 32 mutates the meme unit based on the user behavior data to obtain a multimodal dynamic meme, which specifically includes: the processing module 32 determines the initial meme logic based on the meme unit; the processing module 32 mutates the initial meme logic according to the user behavior data to obtain the target meme logic; the processing module 32 sends the target meme logic to the target device, and obtains the output data sent by the target device and generated by the target device after processing according to the target meme logic; the processing module 32 compares the output data with the actual data to obtain a comparison result.

[0062] In one possible implementation, the processing module 32 mutates the meme unit based on the user behavior data to obtain a multimodal dynamic meme, which specifically includes: the processing module 32 uses the meme fitness function to calculate and obtain a feedback data group, and the feedback data group includes a task completion rate, user click data, and delay data; the processing module 32 performs logical correction on the feedback data group by adjusting the fitness corresponding to the meme fitness function to obtain a correction result; if the processing module 32 determines that the comparison result indicates that the output data is consistent with the actual data and the correction result is normal, the meme unit is assigned weights through the attention mechanism to generate a multimodal dynamic meme.

[0063] In a possible implementation, the processing module 32 migrates multimodal dynamic memes based on the user device to obtain a personalized meme pool, which specifically includes: the processing module 32 obtains the resource status corresponding to the user device; if the processing module 32 determines that the resource status corresponding to the user device does not meet the usage conditions, the processing module 32 obtains the associated device of the user device; the processing module 32 sends the multimodal dynamic memes to the associated device; the processing module 32 determines the personalized meme pool based on the meme unit corresponding to the user device and the multimodal dynamic memes corresponding to the associated device.

[0064] In a possible implementation, the acquisition module 31 acquires usage data of the user when using the smart home device; the processing module 32 inputs the usage data into the multi-terminal collaborative model to obtain the target meme group; the processing module 32 collaboratively controls the smart home device according to the target meme group.

[0065] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0066] The present application also provides an electronic device, referring to Figure 4 , Figure 4 The electronic device may include: at least one processor 41 , at least one network interface 44 , a user interface 43 , a memory 45 , and at least one communication bus 42 .

[0067] The communication bus 42 is used to realize the connection and communication between these components.

[0068] The user interface 43 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 43 may also include a standard wired interface and a wireless interface.

[0069] The network interface 44 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0070] Among them, the processor 41 may include one or more processing cores. The processor 41 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 45, and calling data stored in the memory 45. Optionally, the processor 41 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 41 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 41, and it can be implemented separately through a chip.

[0071] Among them, the memory 45 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 45 includes a non-transitory computer-readable storage medium. The memory 45 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 45 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 45 may also be optionally at least one storage device located away from the aforementioned processor 41. As Figure 4 As shown, the memory 45 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program for a method for constructing a multi-terminal collaborative large model.

[0072] exist Figure 4In the electronic device shown, the user interface 43 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 41 can be used to call an application program for building a multi-terminal collaborative large model stored in the memory 45. When executed by one or more processors, the electronic device executes one or more methods such as those in the above-mentioned embodiments.

[0073] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0074] The present application also provides a computer-readable storage medium, which stores instructions. When executed by one or more processors, the electronic device executes one or more of the methods described in the above embodiments.

[0075] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0077] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0078] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0080] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, it will be easy for those skilled in the art to think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for constructing a multi-terminal collaborative large model, characterized in that: The method comprises: Get the data to be processed sent by the device group; Generate a meme unit using a meme generator according to the data to be processed; Acquire user behavior data corresponding to a user device from the data to be processed, the device group including the user device; Based on the user behavior data, the meme unit is mutated to obtain a multimodal dynamic meme; Based on the user device, migrating the multimodal dynamic meme to obtain a personalized meme pool; Based on the personalized meme pool, a multi-terminal collaborative large model is constructed.

2. The method for constructing a multi-terminal collaborative large model according to claim 1, characterized in that: The obtaining of the data to be processed sent by the device group specifically includes: Acquire visual data sent by an image acquisition device; Get the voice data sent by the voice collection device; Acquire the user behavior data sent by the user device; The visual data, the voice data and the user behavior data are processed to obtain the data to be processed, wherein the data processing includes denoising, filtering and normalization processing.

3. The method for constructing a multi-terminal collaborative large model according to claim 1, characterized in that: The step of generating a meme unit by using a meme generator according to the data to be processed specifically includes: By means of the meme generator, data features corresponding to the data to be processed are mapped to a pre-trained universal meme pool, and a task type is determined, wherein the task type includes image classification, speech recognition, and user behavior recognition; Based on the task type, the meme unit is activated.

4. The method for constructing a multi-terminal collaborative large model according to claim 1, characterized in that: The step of mutating the meme unit based on the user behavior data to obtain a multimodal dynamic meme specifically includes: Based on the meme unit, determining an initial meme logic; According to the user behavior data, the initial meme logic is mutated to obtain a target meme logic; Sending the target meme logic to a target device, and acquiring output data sent by the target device and generated after the target device processes the data according to the target meme logic; The output data is compared with the actual data to obtain a comparison result.

5. The method for constructing a multi-terminal collaborative large model according to claim 4, characterized in that: The method of mutating the meme unit based on the user behavior data to obtain a multimodal dynamic meme specifically includes: A feedback data set is obtained by calculating the memetic fitness function, wherein the feedback data set includes a task completion rate, user click data, and delay data; By adjusting the fitness corresponding to the meme fitness function, the feedback data group is logically corrected to obtain a correction result; If it is determined that the comparison result indicates that the output data is consistent with the actual data, and the correction result is normal, weights are assigned to the meme units through an attention mechanism to generate the multimodal dynamic meme.

6. The method for constructing a multi-terminal collaborative large model according to claim 1, characterized in that: The step of migrating the multimodal dynamic meme based on the user device to obtain a personalized meme pool specifically includes: Obtaining a resource status corresponding to the user equipment; If it is determined that the resource status corresponding to the user equipment does not meet the usage condition, obtaining an associated device of the user equipment; sending the multimodal dynamic meme to the associated device; The personalized meme pool is determined based on the meme unit corresponding to the user device and the multimodal dynamic meme corresponding to the associated device.

7. The method for constructing a multi-terminal collaborative large model according to claim 1, characterized in that: The method further comprises: Obtain usage data of users when using smart home devices; Inputting the usage data into the multi-terminal collaborative large model to obtain a target meme group; The smart home devices are collaboratively controlled according to the target meme group.

8. A device for constructing a multi-terminal collaborative large model, characterized in that: The construction device comprises an acquisition module (31) and a processing module (32), wherein: The acquisition module (31) is used to acquire the data to be processed sent by the device group; The processing module (32) is used to generate a meme unit using a meme generator according to the data to be processed; The acquisition module (31) is further used to acquire user behavior data corresponding to the user device from the data to be processed, and the device group includes the user device; The processing module (32) is further used to mutate the meme unit based on the user behavior data to obtain a multimodal dynamic meme; The processing module (32) is further used to migrate the multimodal dynamic meme based on the user device to obtain a personalized meme pool; The processing module (32) is also used to construct a multi-terminal collaborative large model based on the personalized meme pool.

9. An electronic device, characterized in that: The electronic device comprises a processor (41), a memory (45), a user interface (43) and a network interface (44), wherein the memory (45) is used to store instructions, the user interface (43) and the network interface (44) are both used to communicate with other devices, and the processor (41) is used to execute the instructions stored in the memory (45) so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is performed.

Images