Multimodal data processing method and apparatus, child camera, and storage medium

Abstract

The embodiment of the application relates to the technical field of data processing, and particularly discloses a multi-modal data processing method and device, a child camera and a storage medium, after multi-modal data is acquired, a target function mode is automatically determined, the multi-modal data and function parameters are encapsulated into a first structure body in a unified format for unified routing processing, and a processing result is encapsulated into a second structure body in a unified format and returned. The embodiment of the application makes different function modes share a unified data inlet and outlet, only needs to set a unified data docking and result analysis logic for the first structure body and the second structure body, does not need to separately adapt interfaces for each function, only needs to increase a corresponding processing function when a new function mode is added, and does not need to modify existing codes, so that the development and maintenance cost of function expansion is reduced, and the efficiency and uniformity of multi-modal data processing are improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a multimodal data processing method, apparatus, children's camera and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, smart terminal devices such as children's cameras have gradually integrated a variety of AI-based functions, such as object recognition, question answering, and image generation, to meet the cognitive and creative needs of children.

[0003] Currently, children's cameras typically have separate entry points for functions such as object recognition, question-and-answer, and image generation, and the data processing flows for each function are independent: the photo-taking function only receives image data, the question-and-answer function only receives voice data, and the processing results of each function are returned in their own formats. Because the input data formats and output result formats of each function are inconsistent, the application needs to write separate logic for data integration and result parsing for each function, resulting in high code duplication and maintenance costs. Summary of the Invention

[0004] In view of this, embodiments of this application provide a multimodal data processing method, apparatus, children's camera, and storage medium, which can improve the efficiency and uniformity of multimodal data processing by children's cameras.

[0005] In a first aspect, embodiments of this application provide a multimodal data processing method applied to a children's camera, comprising the following steps: acquiring multimodal data and determining a target functional mode based on the multimodal data; encapsulating the multimodal data and the parameters of the target functional mode into a first structure in a unified format; determining a target processing function based on the target functional mode; executing the target processing function based on the first structure to obtain a processing result; and encapsulating the processing result into a second structure in a unified format.

[0006] In some embodiments, the step of determining the target functional mode based on the multimodal data further includes: extracting features from the multimodal data to obtain feature information; and determining the target functional mode based on the feature information.

[0007] In some embodiments, the method further includes: determining the type of the multimodal data; determining whether the target functional mode corresponds to the type of the multimodal data; if the target functional mode corresponds to the type of the multimodal data, then determining the target functional mode; if the target functional mode does not correspond to the type of the multimodal data, then determining a new target functional mode based on the type of the multimodal data.

[0008] In some embodiments, the first structure includes at least an input data type field, an input data feature field, a function mode field, a function parameter field, and a first reserved extended field.

[0009] In some embodiments, the second structure includes at least a result type field, a result data feature field, a processing status field, and a second reserved extended field.

[0010] In some embodiments, the processing status field is used to record a return code and a task identifier. The return code is used to characterize the processing progress of the target processing function, and the task identifier is used to asynchronously query the processing result of the target processing function.

[0011] In some embodiments, the method further includes: when the return code indicates that the target processing function has not completed processing, querying the processing result again according to the task identifier until a return code indicating that the target processing function has completed processing is obtained.

[0012] In some embodiments, the method further includes: generating an output control signal based on the second structure, the output control signal being used to control the output of the processing result.

[0013] In some embodiments, the multimodal data includes at least one of images, audio, and text.

[0014] In some embodiments, the target functional modes include at least object recognition mode, text-to-image mode, voice-to-image mode, image style conversion mode, voice question-and-answer mode, and image-text question-and-answer mode.

[0015] In some embodiments, the target processing function is used to invoke a large model to process the multimodal data, wherein the large model includes at least a visual large model, a speech large model, and a multimodal large model.

[0016] Secondly, embodiments of this application provide a multimodal data processing device applied to a children's camera, comprising: an acquisition module for acquiring multimodal data and determining a target functional mode based on the multimodal data; a first encapsulation module for encapsulating the multimodal data and parameters of the target functional mode into a first structure of a unified format; a determination module for determining a target processing function based on the target functional mode; a processing module for executing the target processing function based on the first structure to obtain a processing result; and a second encapsulation module for encapsulating the processing result into a second structure of a unified format.

[0017] Thirdly, embodiments of this application provide a child camera, comprising: at least one processor; at least one memory communicatively connected to the processor, the memory being used to store at least one program; when at least one program is executed by at least one processor, the at least one processor implements the above-described method.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program or instructions, which, when executed, implement the above-described method.

[0019] The beneficial effects of this application's implementation are as follows: Unlike existing technologies, this application provides a multimodal data processing method applied to children's cameras. After acquiring multimodal data, it automatically determines the target functional mode and encapsulates the multimodal data and functional parameters into a unified first structure for unified routing processing. The processing result is then encapsulated into a unified second structure and returned. This application's embodiment enables different functional modes to share a unified data entry and exit point. Only a unified data interface and result parsing logic needs to be set for the first and second structures, eliminating the need for separate interface adaptations for each function. When adding a new functional mode, only the corresponding processing function needs to be added, without modifying existing code. This reduces the development and maintenance costs of functional expansion and improves the efficiency and uniformity of multimodal data processing. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the drawings without creative effort.

[0021] Figure 1 This is a schematic diagram illustrating an application scenario of a multimodal data processing method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a multimodal data processing method provided in an embodiment of this application; Figure 3 yes Figure 1 A schematic diagram of a sub-process of step S10 in the multimodal data processing method shown; Figure 4 This is a schematic diagram of the structure of a multimodal data processing device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a children's camera provided in an embodiment of this application. Detailed Implementation

[0022] To facilitate understanding of this application, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. It should be noted that, unless otherwise specified, the various features in the embodiments of this application can be combined with each other, all within the scope of protection of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish between identical or similar items with essentially the same function and effect.

[0023] Please see Figure 1 , Figure 1 This application scenario describes a multimodal data processing method provided in this embodiment, including a terminal device 10 and a server 20. The terminal device 10 and the server 20 communicate via a network, such as a local area network (LAN), wide area network (WAN), wireless network, Global System for Mobile Communications (GSM), third-generation mobile communication network, fourth-generation mobile communication network, and fifth-generation mobile communication network.

[0024] Terminal device 10 can be a children's camera. In some embodiments, terminal device 10 has the ability to acquire multimodal data, for example, terminal device 10 can acquire at least one type of multimodal data, including images, voice, and text; terminal device 10 also has data processing capabilities, capable of executing the multimodal data processing method provided in the embodiments of this application, for example, determining a target functional mode based on the acquired multimodal data, encapsulating the multimodal data and the parameters of the target functional mode into a first structure of a unified format, determining a target processing function based on the target functional mode, and encapsulating the processing result into a second structure of a unified format, etc.; terminal device 10 also has the ability to output results, for example, generating an output control signal based on the second structure to control the output processing result, such as displaying text or images, playing voice, etc.

[0025] Server 20 can be a single server, such as a rack server, blade server, tower server, or cabinet server, or a server cluster consisting of several servers, or a cloud computing service center. In some embodiments, server 20 has artificial intelligence-based data processing capabilities. For example, server 20 is equipped with at least one of the following: processing capabilities for image recognition or understanding, processing capabilities for speech-to-text conversion, and processing capabilities for image generation. It can respond to processing requests sent by terminal device 10, process multimodal data, and return the corresponding processing results to terminal device 10.

[0026] In some embodiments, the terminal device 10 and the server 20 cooperate to complete the multimodal data processing method of this application embodiment: after the terminal device 10 acquires multimodal data, it determines the target functional mode based on the multimodal data, encapsulates the multimodal data and the parameters of the target functional mode into a first structure in a unified format, determines the target processing function based on the target functional mode, and sends the processing request to the server 20 through the network; the server 20 processes the multimodal data and returns the processing result; the terminal device 10 receives the processing result and encapsulates the processing result into a second structure in a unified format.

[0027] In other embodiments, the terminal device 10 has the above-mentioned data processing capabilities based on artificial intelligence. The multimodal data processing method of this application embodiment can also be executed independently by the terminal device 10 without communicating with the server 20. This application embodiment does not limit this. It should be noted that the method provided in this application embodiment can be further extended to other suitable application environments, and is not limited to the application environments described above. In actual application, the application environment may also include more or fewer servers and terminal devices.

[0028] Please see Figure 2 , Figure 2 This is a flowchart illustrating a multimodal data processing method provided in an embodiment of this application.

[0029] Multimodal data processing methods are applied to children's cameras, such as Figure 1 As shown, the method includes the following steps: Step S10: Acquire multimodal data and determine the target functional mode based on the multimodal data.

[0030] Specifically, multimodal data includes at least one of images, audio, and text. For example, child users obtain image data by taking pictures, audio data by recording audio, and text data by touch or keyboard input.

[0031] In some embodiments, multimodal data also includes identification data generated by user input operations, such as by pressing function keys or touching function icons. Multimodal data is not limited to the above forms and may also include data of other modalities, such as video data, gesture data, etc.; multimodal data can be one type of data as described above, or a combination of multiple types of data, such as a combination of images and text, a combination of voice and text, or a combination of images and identification data.

[0032] Among them, children's cameras can directly identify the types of images, audio, and text in multimodal data. For example, the type can be determined based on the source of the multimodal data collection, such as data collected by a camera being images and data collected by a microphone being audio; or, the type can be determined based on the file extension or file header of the multimodal data, such as files with extensions of .jpg or .png being images, files with extensions of .wav or .mp3 being audio, and files with extensions of .txt being text.

[0033] The target functional modes include at least object recognition mode, text-to-image mode, voice-to-image mode, image style conversion mode, voice question-and-answer mode, and image-text question-and-answer mode. Specifically, object recognition mode identifies objects in an image and outputs the recognition result; text-to-image mode generates an image based on text; voice-to-image mode generates an image based on speech; image style conversion mode performs stylization processing on the image; voice question-and-answer mode outputs an answer based on a voice question; and image-text question-and-answer mode outputs an answer based on both an image and text question. It should be noted that the target functional modes are not limited to the above examples, and other functional modes can be extended according to product needs; this application embodiment does not limit this.

[0034] In some embodiments, please refer to the following: Figure 3 , Figure 3 yes Figure 2 A sub-process of step S10 in the method. Determining the target functional mode based on multimodal data, further includes: Step S11: Extract features from the multimodal data to obtain feature information.

[0035] Specifically, feature information includes content information. Content information refers to the content extracted from images, audio, or text, such as the text content obtained by speech recognition of audio data, the text content of the text data itself, or the scene and object information obtained by content recognition of image data.

[0036] In some embodiments, the feature information includes a pattern identifier, which is identification data generated by the user's input operation to identify the functional mode. For example, each function button or function icon of a children's camera corresponds one-to-one with a functional mode. When the user presses the object recognition button or touches the object recognition icon, the children's camera detects the input operation and generates identification data corresponding to the object recognition mode, thus obtaining the pattern identifier indicating the object recognition mode.

[0037] Step S12: Determine the target functional mode based on the feature information.

[0038] Specifically, in some embodiments, when the feature information includes a pattern identifier, the functional mode identified by the pattern identifier is determined as the target functional mode based on a preset correspondence between the pattern identifier and the functional mode. For example, when a user presses the object recognition button to take a picture of a flower, the children's camera obtains the pattern identifier indicating the object recognition mode and determines the object recognition mode as the target functional mode.

[0039] In some embodiments, when the feature information includes content information, it is identified whether the content information contains information representing the intention to select a function; if so, the target function mode is determined based on the intention to select a function. For example, when the intention to generate an image is identified in the voice or text content, such as "draw a cat", the target function mode is determined to be text-to-image mode or voice-to-image mode; when the intention to identify an object is identified in the content, such as "what kind of flower is this", the target function mode is determined to be object recognition mode or image-text question-and-answer mode; when the intention to stylize is identified in the content, such as "turn this photo into a cartoon style", the target function mode is determined to be image style conversion mode; when the intention to ask questions and interact is identified in the content, such as "why is the sky blue", the target function mode is determined to be voice question-and-answer mode or image-text question-and-answer mode.

[0040] In some embodiments, when neither the feature information nor the content information contains a function selection intent, the target function mode is determined based on the type of multimodal data. The child camera is pre-configured with a mapping relationship between data types and function modes. For example, image types map to object recognition mode, voice types map to voice question-and-answer mode, text types map to text-to-image mode, and mixed image and text types map to text-to-image question-and-answer mode. For instance, if a child user takes a photo without any input, and the child camera does not obtain a pattern identifier and does not recognize a function selection intent from the photo data, then the object recognition mode is determined as the target function mode based on the image type.

[0041] Therefore, the children's camera can be compatible with explicit user input, implicit intentions expressed in voice or text, and default situations without any instructions. It can automatically determine the matching target function mode under various usage methods, without requiring users to understand the structure of the function menu, thus reducing the operation steps for function switching.

[0042] Step S13: Determine the type of multimodal data and determine whether the target functional mode corresponds to the type of multimodal data.

[0043] Specifically, the type of multimodal data is directly identified and confirmed by the children's camera, including images, voice, text, and combinations of multiple types. In some embodiments, the children's camera is pre-configured with a compatibility matrix, which represents the correspondence between each functional mode and each data type. For example, the object recognition mode corresponds to the image type; the image-text question-and-answer mode corresponds to the image type, text type, and mixed image-text type; the text-to-image mode corresponds to the text type; the voice-to-image mode corresponds to the voice type; the image style conversion mode corresponds to the image type; and the voice question-and-answer mode corresponds to the voice type. By querying the compatibility matrix, it can be determined whether the target functional mode determined in step S12 corresponds to the type of multimodal data.

[0044] Step S14: If the target functional mode corresponds to the type of multimodal data, then determine the target functional mode.

[0045] Step S15: If the target functional mode does not correspond to the type of multimodal data, then determine a new target functional mode based on the type of multimodal data.

[0046] Specifically, when the target function mode does not correspond to the type of multimodal data, the system automatically switches according to the type of multimodal data, determining the function mode corresponding to the data type as the new target function mode, and can output a prompt message to inform the user that the automatic switch has been completed. For example, after pressing the object recognition button, the user says "What animal is this?" into the microphone, but no photo is taken, and only voice data is input; step S12 determines the target function mode as the object recognition mode based on the mode identifier. Since the object recognition mode corresponds to the image type but not the voice type, the children's camera automatically determines the voice question-and-answer mode as the new target function mode based on the voice type, responding to the user's question in a voice question-and-answer manner. Thus, when the function selected by the user does not match the input data, it can be automatically corrected, avoiding invalid processing requests due to function and input mismatch, reducing the uploading of invalid data and the waste of processing resources.

[0047] Step S20: Encapsulate the multimodal data and the parameters of the target functional mode into a first structure in a unified format.

[0048] Specifically, the first structure includes at least an input data type field, an input data feature field, a function mode field, a function parameter field, and a first reserved extended field.

[0049] The input data type field is used to record the type of multimodal data. In some examples, the input data type field records the data type using enumerated values. For example, the first enumerated value represents the image type, the second enumerated value represents the speech type, the third enumerated value represents the text type, the fourth enumerated value represents the combination of image and text, and the fifth enumerated value represents the combination of speech and text. By recording the data type, including combination types, using enumerated values, mixed-input multimodal data can be carried in a unified format by the first structure.

[0050] The input data feature field is used to carry multimodal data or feature information corresponding to the data type. In some embodiments, the input data feature field includes multiple subfields corresponding to various data types. For example, the image path subfield records the storage path of image data, the voice path subfield records the storage path of voice data, and the text subfield records the text content. The child camera reads the data from the corresponding subfield according to the type recorded in the input data type field. By using storage paths to carry large amounts of data such as images and voice, the copying and moving of data entities is avoided, reducing memory usage and data transmission overhead during the encapsulation process.

[0051] The Function Mode field records the target function mode. The Function Parameter field carries the function parameters corresponding to the target function mode. In some embodiments, the function parameters include at least one of a role identifier, a language identifier, a generated image size, and a style type. The role identifier indicates the tone of voice used in the question-and-answer or object recognition result, such as a doctor character or a cartoon character; the language identifier indicates the language of the output, such as Chinese, English, or other languages; the generated image size indicates the width and height of the generated image; and the style type indicates the target style for style conversion, such as cartoon style or watercolor style. In some embodiments, before encapsulating the first structure, the children's camera completes the missing function parameters with default parameters based on the target function mode: when the target function mode is an object recognition mode, a text-to-image question-and-answer mode, or a voice question-and-answer mode, it completes the default role identifier and default language identifier; when the target function mode is a text-to-image mode, a voice-to-image mode, or an image style conversion mode, it completes the default generated image size. Therefore, users only need to provide multimodal data to obtain the processing results, further reducing user input.

[0052] The first reserved extended field is used to carry new functional parameters when adding a new functional mode. In some examples, the first reserved extended field includes an extended data pointer and an extended data length. The extended data pointer points to the storage address of the extended parameter, and the extended data length records the data length of the extended parameter. Through the pointer plus length reservation mechanism, the new parameters required by the new functional mode can be attached to the first reserved extended field without changing the existing field definitions of the first structure. The parsing logic of the existing fields by the caller and the callee is not affected, ensuring the forward compatibility of the framework.

[0053] It should be noted that the first structure may also be or include one or more other fields, and this application embodiment does not limit this.

[0054] In some embodiments, before encapsulating the first structure, default parameters are added to complete the missing functional parameters according to the target functional mode. For example, when the target functional mode is object recognition mode, text-to-image question-and-answer mode, or voice question-and-answer mode, default role identifiers and default language identifiers are added; when the target functional mode is text-to-image mode, voice-to-image mode, or image style conversion mode, default generated image size is added. Thus, users only need to provide multimodal data to obtain processing results, further reducing user input.

[0055] Step S30: Determine the target processing function based on the target functional mode.

[0056] Specifically, the children's camera pre-configures a correspondence between various function modes and processing functions. By querying this correspondence based on the target function mode, the target processing function corresponding to the target function mode can be determined. In some embodiments, a branch routing structure is used to implement the above determination process, that is, based on the value of the target function mode, the system jumps to the corresponding branch to call the corresponding processing function. When a new function mode is added, only the branch and processing function corresponding to the new function mode need to be added to the branch routing structure, without modifying the existing branches and processing functions, thereby achieving rapid expansion of functionality.

[0057] The target processing function is used to call a large model to process multimodal data. The large model includes at least a visual large model, a speech large model, and a multimodal large model. Specifically, the visual large model is used for image recognition or understanding; the speech large model is used for speech-to-text conversion, such as automatic speech recognition and text-to-speech; and the multimodal large model is used to generate images based on text or speech. It should be noted that the large model can also be other models capable of processing multimodal data. This application does not limit the network structure, model size, or training method of the large model.

[0058] In some embodiments, a functional mode can invoke multiple large models for collaborative processing. For example, in the speech-to-image mode, the target processing function first invokes the speech large model to convert speech into text, and then invokes the multimodal large model to generate an image based on the text. A single large model can also serve multiple functional modes; for example, the visual large model serves both the object recognition mode and the image-to-text question-and-answer mode and the image style conversion mode. The target processing function determines the large models to be invoked and the invocation order based on the processing requirements of the target functional mode.

[0059] Step S40: Execute the target processing function based on the first structure to obtain the processing result.

[0060] Specifically, when the target processing function is executed, it reads multimodal data and functional parameters from the first structure, calls the corresponding processing resources to process the multimodal data, and obtains the processing result. For example, in object recognition mode, the target processing function reads image data from the first structure, calls the visual large model to recognize the image, and obtains a processing result containing recognized text and recognized speech; in text-to-image mode, the target processing function reads text content and image size parameters from the first structure, calls the multimodal large model to generate an image, and obtains a processing result containing the image file address.

[0061] Because some functional modes are time-consuming to process, in some embodiments, the child camera does not block and wait after sending a processing request to the processing resource, but instead obtains the processing result asynchronously. Specifically, after executing the target processing function, the processing result is obtained. The processing result can be the final processing result or an intermediate result that is not yet fully processed. The processing result returned by the processing resource can include a return code and a task identifier. Each time the child camera initiates a processing request, the processing resource assigns a corresponding task identifier to the processing request. The task identifier is used to identify the processing task corresponding to the processing request. When there are multiple concurrent processing tasks, each processing task corresponds to a different task identifier. The child camera can asynchronously query the corresponding processing result of the target processing function through the task identifier. The return code is used to characterize the processing progress of the target processing function.

[0062] In some embodiments, the method further includes: when a return code indicates that the target processing function has not completed processing, querying the processing result again based on the task identifier until a return code indicating that the target processing function has completed processing is obtained. In some examples, the processing result is queried periodically at preset time intervals; in other examples, the processing result is queried based on the task identifier after receiving a result-ready notification sent by the processing resource.

[0063] Step S50: Encapsulate the processing result into a second structure in a unified format.

[0064] Specifically, the second structure includes at least a result type field, a result data feature field, a processing status field, and a second reserved extended field.

[0065] The result type field records the type of the processing result. Result types include single types such as text, image, and voice, as well as combinations of multiple types, such as a combination of text and voice, or a combination of image and text. In some examples, the result type field also records the result type using enumeration values. By representing result types, including combined types, with a unified enumeration system, upper-layer applications only need to determine the parsing method based on the result type field.

[0066] The result data feature field is used to carry the content of the processing result. In some embodiments, the result data feature field includes a text result subfield and a file address subfield: the text result subfield carries an array of text results and the number of texts, and the text result array may include multiple text results; the file address subfield carries an array of file addresses and the number of files, and each file address in the file address array indicates the location where the image file or audio file is obtained, such as a Uniform Resource Locator (URL). By setting the array and the number in pairs, the processing result can include multiple texts and multiple files, and the upper-layer application can traverse and parse according to the number, without needing to pre-define the number of result entries.

[0067] The processing status field records return codes and task identifiers. Return codes characterize the processing progress of the target processing function. In some examples, return codes include at least a first return code indicating processing is in progress and a second return code indicating processing is complete. They may also include return codes indicating processing failure or no valid task. The task identifier is used to asynchronously query the processing result of the target processing function. All processing results are returned through a unified formatted second structure. Upper-layer applications only need to determine whether to read the processing result based on the processing status field; there is no need to write separate processing logic for synchronous and asynchronous scenarios.

[0068] The second reserved extension field is used to hold the extended result data. In some examples, the second reserved extension field uses the same pointer plus length mechanism as the first reserved extension field. The extended result data generated by the new feature mode can be attached to the second reserved extension field without changing the existing field definitions of the second structure.

[0069] In some embodiments, the method further includes: step S60: generating an output control signal based on the second structure, wherein the output control signal is used to control the output processing result.

[0070] Specifically, in some embodiments, the processor or controller of the children's camera parses the result type field of the second structure and automatically generates corresponding output control signals based on the type of the processing result. For example: when the result type field indicates that the processing result includes audio, an audio download instruction is generated. This instruction triggers a download task to download the corresponding audio file and controls the audio playback unit to play the downloaded audio file; when the result type field indicates that the processing result includes an image, a preview instruction is generated. This instruction triggers the download of a preview image and controls the display interface to display the preview image; when the result type field indicates that the processing result includes text, a display instruction is generated. This instruction controls the display interface to display the corresponding text. Thus, based on the output control signals, download, display, and playback actions are automatically executed. Upper-layer applications do not need to write separate output trigger logic for different result types, further reducing repetitive code and improving the output efficiency of the processing results.

[0071] The output action is executed by the corresponding component of the child camera in response to the output control signal. For example, in some embodiments, the download module of the processor or controller, in response to an audio download instruction, downloads the corresponding audio file according to the audio file address recorded in the result data feature field of the second structure; after the download is completed, for example, the audio playback unit of the speaker, in response to the audio download instruction, plays the downloaded audio file. In some examples, the download module performs the download of the audio file as a separate download task, and the download process does not block other processing flows of the child camera; after the download is completed, the download module sends a playback notification to the audio playback unit, and the audio playback unit performs playback in response to the playback notification.

[0072] For example, in a photo-based object recognition scenario, a child user presses the object recognition button to take a picture of a flower. The child camera obtains the image data and the identifier data indicating the object recognition mode. Based on the mode identifier, the object recognition mode is determined as the target function mode. After compatibility verification confirms that the object recognition mode corresponds to the image type, the default role identifier and default language identifier are completed. The photo data and function parameters are encapsulated into a first structure and routed to the processing function corresponding to the object recognition mode. This processing function calls the visual big model to recognize the photo. The child camera encapsulates the recognized text and voice into a second structure and generates display instructions and audio download instructions based on the result type field. It controls the display screen to display the recognized text and controls the speaker to play the recognized voice, explaining the flower in the photo to the child in a role-playing tone.

[0073] In this embodiment, the multimodal data processing method encapsulates multimodal data and parameters of the target functional mode into a unified first structure for unified routing processing, and returns the processing result encapsulated into a unified second structure, enabling different functional modes to share a unified data entry and exit point. Upper-layer applications only need to set up a unified set of data interface and result parsing logic for the first and second structures, without needing to adapt interfaces separately for each function. When a new functional mode is added, only the processing function corresponding to the new functional mode needs to be added, without modifying the code of the upper-layer application or existing functional modes, thereby reducing the development and maintenance costs of functional expansion and improving the scalability and reliability of multimodal data processing.

[0074] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a multimodal data processing device provided in an embodiment of this application.

[0075] like Figure 4 As shown, the multimodal data processing device 100 applied to children's cameras includes an acquisition module 101, a first encapsulation module 102, a determination module 103, a processing module 104, and a second encapsulation module 105.

[0076] The acquisition module 101 acquires multimodal data and determines the target functional mode based on the multimodal data. The first encapsulation module 102 encapsulates the multimodal data and the parameters of the target functional mode into a first structure with a unified format. The determination module 103 determines the target processing function based on the target functional mode. The processing module 104 executes the target processing function based on the first structure to obtain the processing result. The second encapsulation module 105 encapsulates the processing result into a second structure with a unified format.

[0077] In some embodiments, the multimodal data processing apparatus 100 further includes an output control module for generating an output control signal based on the second structure.

[0078] In this embodiment, the multimodal data processing device 100 can be a software module. The software module includes several instructions, which are stored in a memory. The processor can access the memory and call the instructions to execute them, so as to complete the multimodal data processing methods of the above embodiments.

[0079] In the embodiments of this application, the multimodal data processing device 100 can also be constructed from hardware devices. For example, the multimodal data processing device 100 can be constructed from one or more chips, and the chips can work together to complete the multimodal data processing methods described in the above embodiments. As another example, the multimodal data processing device 100 can also be constructed from various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM (Acorn RISC Machine) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0080] The multimodal data processing device 100 in this embodiment can be a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This embodiment does not impose specific limitations.

[0081] The multimodal data processing device 100 in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit its use.

[0082] The multimodal data processing apparatus 100 provided in this application embodiment can implement all the processes implemented in the method embodiment of this application, and will not be described again here to avoid repetition.

[0083] It should be noted that the above-described apparatus can execute the multimodal data processing method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in the apparatus embodiments can be found in the multimodal data processing method provided in the embodiments of this application.

[0084] In this embodiment, the efficiency and uniformity of the children's camera in processing multimodal data can be improved by the cooperation of the various modules of the multimodal data processing device 100.

[0085] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a children's camera provided in an embodiment of this application.

[0086] like Figure 5 As shown, the child camera 200 includes at least one processor 21 and at least one memory 22, the at least one memory 22 being used to store at least one program. When the at least one program is executed by the at least one processor 21, the at least one processor 21 implements the multimodal data processing method of any method embodiment of this application.

[0087] The processor 21 and the memory 22 can be connected via a bus or other means. The processor 21 is used to execute the multimodal data processing method in any embodiment of this application.

[0088] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the multimodal data processing method in this embodiment of the invention. The processor 21 executes various functional applications and data processing of the child camera 200 by running the non-volatile software programs, instructions, and modules stored in the memory 22, thereby implementing the multimodal data processing method of this application embodiment.

[0089] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21. Examples of the above-described networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0090] The multimodal data processing method of any method embodiment of this application can be divided into one or more functional modules and stored in the memory 22. When executed by one or more processors 21, the multimodal data processing method of any method embodiment of this application is executed.

[0091] In some embodiments, the processor 21 and memory 22 are integrated into the controller or chip.

[0092] In some embodiments, the children's camera 200 further includes a data acquisition component 23, an input component 24, an output component 25, and a communication component 26, which are respectively connected to the processor 21 via a bus or other means.

[0093] Acquisition unit 23 is used to acquire multimodal data, including, for example, a camera and a microphone. The camera is used to acquire image data, and the microphone is used to acquire voice data. Input unit 24 is used to receive user input operations, including, for example, at least one of function keys and a touch screen. The user's input operations on function keys or function icons on the touch screen are converted into identification data that identifies the function mode. Output unit 25 is used to output processing results, including, for example, a display screen and a speaker. The display screen is used to display text or images from the processing results, and the speaker is used to play the voice from the processing results.

[0094] The communication component 26 is used to communicate with the server 20. For example, when the processor 21 executes the target processing function, it sends a processing request to the server 20 through the communication component 26 and receives the processing result returned by the server 20. In asynchronous processing scenarios, the processor 21 also sends a query request carrying a task identifier to the server 20 through the communication component 26 to obtain the processing progress and processing result. The communication component 26 can be a wireless communication module, such as one or more of a WIFI module, Bluetooth module, fourth-generation mobile communication module, or fifth-generation mobile communication module.

[0095] In some embodiments, the child camera 200 further includes a power supply 18, which is connected to the aforementioned components and is used to supply power to the components.

[0096] It should be noted that, Figure 5 The structure of the child camera 200 shown is for illustrative purposes only. The child camera 200 may also include more or fewer components than shown, or combine certain components, or have different component arrangements. This application embodiment does not limit this.

[0097] This application also provides a computer-readable storage medium storing a computer program or instructions that, when executed, implement the multimodal data processing method described in any of the above method embodiments. For example, one or more processors may execute the multimodal data processing method described in any of the above method embodiments, or execute the steps described above.

[0098] The apparatus or device embodiments described above are merely illustrative. The unit modules described as separate components may or may not be physically separate, and the components shown as module units may or may not be physical units; that is, they may be located in one place or distributed across multiple network module units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions for a computer device (which may be a personal computer, server, or network device, etc.) to execute the various embodiments or some parts of the embodiments.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; under the concept of the present invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal data processing method applied to children's cameras, characterized in that, Includes the following steps: Acquire multimodal data and determine the target functional mode based on the multimodal data; The multimodal data and the parameters of the target functional mode are encapsulated into a first structure in a unified format; Based on the target functional mode, determine the target processing function; The target processing function is executed based on the first structure to obtain the processing result; The processing result is encapsulated into a second structure in a unified format.

2. The method according to claim 1, characterized in that... , The step of determining the target functional mode based on the multimodal data further includes: Feature extraction is performed on the multimodal data to obtain feature information; Based on the aforementioned feature information, the target functional mode is determined.

3. The method according to claim 2, characterized in that, The method further includes: Determine the type of the multimodal data and determine whether the target functional mode corresponds to the type of the multimodal data; If the target functional mode corresponds to the type of the multimodal data, then the target functional mode is determined; If the target functional mode does not correspond to the type of the multimodal data, then a new target functional mode is determined based on the type of the multimodal data.

4. The method according to claim 1, characterized in that, The first structure includes at least an input data type field, an input data feature field, a function mode field, a function parameter field, and a first reserved extended field.

5. The method according to claim 1, characterized in that, The second structure includes at least a result type field, a result data feature field, a processing status field, and a second reserved extended field.

6. The method according to claim 5, characterized in that, The processing status field is used to record the return code and the task identifier. The return code is used to characterize the processing progress of the target processing function, and the task identifier is used to asynchronously query the processing result of the target processing function.

7. The method according to claim 6, characterized in that, The method further includes: When the return code indicates that the target processing function has not completed processing, the processing result is queried again according to the task identifier until a return code indicating that the target processing function has completed processing is obtained.

8. The method according to claim 7, characterized in that, The method further includes: An output control signal is generated based on the second structure, and the output control signal is used to control the output of the processing result.

9. The method according to any one of claims 1-8, characterized in that, The multimodal data includes at least one of images, audio, and text; and / or, The target functional modes include at least object recognition mode, text-to-image mode, voice-to-image mode, image style conversion mode, voice question-and-answer mode, and text-to-image question-and-answer mode; and / or, The target processing function is used to call a large model to process the multimodal data. The large model includes at least a visual large model, a speech large model, and a multimodal large model.

10. A multimodal data processing device, applied to a children's camera, characterized in that, include: An acquisition module is used to acquire multimodal data and determine the target functional mode based on the multimodal data; The first encapsulation module is used to encapsulate the multimodal data and the parameters of the target functional mode into a first structure in a unified format; The determining module is used to determine the target processing function based on the target functional mode; The processing module is used to execute the target processing function based on the first structure to obtain the processing result; The second encapsulation module is used to encapsulate the processing result into a second structure in a unified format.

11. A children's camera, characterized in that, include: At least one processor; At least one memory, communicatively connected to the processor, the memory being used to store at least one program; When at least one of the programs is executed by at least one of the processors, the at least one of the processors implements the method of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions that, when executed, implement the method as described in any one of claims 1-9.

Images