Method of processing visual data and data processing system

Abstract

This specification provides a visual data processing method and a data processing system. The visual data processing method, applied to a visual data processing unit, includes: receiving a data processing request for target visual data, wherein the data processing request carries target text associated with the target visual data, and the target text is obtained by text processing target speech; using a multimodal data processing model to perform visual processing on the target visual data based on the target text to obtain a first data processing result; using the visual data processing model to perform visual processing on the target visual data to obtain a second data processing result; and obtaining a target data processing result based on the first data processing result and the second data processing result.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a visual data processing method. One or more embodiments of this specification also relate to another visual data processing method, a data processing system, an artificial intelligence wearable device, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the continuous development of artificial intelligence technology, neural network models can be applied to various data processing scenarios, such as processing visual data using neural network models; In the process of using neural network models to process visual data, there is a problem of inaccurate data processing results. Therefore, how to improve the accuracy of data processing results has become an urgent problem to be solved. Summary of the Invention

[0003] In view of the above, embodiments of this specification provide a visual data processing method. One or more embodiments of this specification also relate to another visual data processing method, a data processing apparatus, another data processing apparatus, a data processing system, an artificial intelligence wearable device, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a visual data processing method is provided, applied to a visual data processing unit, comprising: Receive a data processing request for target visual data, wherein the data processing request carries target text associated with the target visual data, and the target text is obtained by text processing the target speech; Using a multimodal data processing model, visual processing is performed on the target visual data based on the target text to obtain a first data processing result; The target visual data is processed using a visual data processing model to obtain a second data processing result, and the target data processing result is obtained based on the first data processing result and the second data processing result.

[0005] According to a second aspect of the embodiments of this specification, a visual data processing apparatus is provided, applied to a visual data processing unit, comprising: The request receiving module is configured to receive a data processing request for target visual data, wherein the data processing request carries target text associated with the target visual data, and the target text is obtained by text processing the target speech. The first data processing module is configured to use a multimodal data processing model to perform visual processing on the target visual data based on the target text to obtain a first data processing result. The second data processing module is configured to use a visual data processing model to perform visual processing on the target visual data to obtain a second data processing result, and to obtain a target data processing result based on the first data processing result and the second data processing result.

[0006] According to a third aspect of the embodiments of this specification, a visual data processing method is provided, applied to a visual data management unit, the method comprising: The target speech is subjected to speech processing to obtain the speech processing result of the target speech, wherein the target speech is speech associated with the target visual data; If, based on the speech processing results, it is determined that the target speech meets the model processing conditions, a data processing request is determined based on the target text corresponding to the target speech, wherein the target text is obtained by performing text processing on the target speech; The data processing request is sent to the visual data processing unit so that the visual data processing unit executes the steps of the above-described visual data processing method.

[0007] According to a fourth aspect of the embodiments of this specification, a visual data processing apparatus is provided, applied to a visual data management unit, the method comprising: The speech processing module is configured to perform speech processing on the target speech to obtain the speech processing result of the target speech, wherein the target speech is speech associated with the target visual data; The request determination module is configured to determine a data processing request based on the target text corresponding to the target speech, provided that the target speech meets the model processing conditions based on the speech processing results; wherein the target text is obtained by performing text processing on the target speech. The request sending module is configured to send the data processing request to the visual data processing unit so that the visual data processing unit performs the steps of the above-described visual data processing method.

[0008] According to a fifth aspect of the embodiments of this specification, a data processing system is provided, the data processing system including a visual data management unit and a visual data processing unit; The visual data management unit is configured to perform speech processing on the target speech to obtain the speech processing result of the target speech, wherein the target speech is speech associated with the target visual data. When it is determined that the target speech meets the model processing conditions based on the speech processing result, a data processing request is determined based on the target text corresponding to the target speech, wherein the target text is obtained by performing text processing on the target speech, and the data processing request is sent to the visual data processing unit. The visual data processing unit is configured to receive a data processing request for the target visual data, wherein the data processing request carries the target text associated with the target visual data, performs visual processing on the target visual data based on the target text using a multimodal data processing model to obtain a first data processing result, performs visual processing on the target visual data using the visual data processing model to obtain a second data processing result, and obtains a target data processing result based on the first data processing result and the second data processing result.

[0009] According to a sixth aspect of the embodiments of this specification, an artificial intelligence wearable device is provided, configured to: Real-time visual data is acquired according to the target's perspective to obtain target visual data, and real-time voice is acquired to obtain target voice. The target's perspective is the same as the human eye's first-person perspective. The target visual data and the target speech are sent to the data processing system, and the target data processing result sent by the data processing system is received. The data processing system is the aforementioned data processing system.

[0010] According to a seventh aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of any of the above methods.

[0011] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of any of the above-described methods.

[0012] According to a ninth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0013] The visual data processing method applied to a visual data processing unit in one or more embodiments of this specification, upon receiving a data processing request for target visual data, utilizes a multimodal data processing model to perform visual processing on the target visual data based on the target text carried in the data processing request, obtaining a first data processing result; and then utilizes a visual data processing model to perform visual processing on the target visual data, obtaining a second data processing result; finally, through the first and second data processing results obtained from the multimodal data processing model and the visual data processing model, an accurate target data processing result is obtained, thus avoiding the problem of inaccurate data processing results. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of an image processing scheme provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating a visual data processing method provided in one embodiment of this specification; Figure 3 This is an execution flowchart of a data processing system provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the interaction process of a visual data processing method provided in one embodiment of this specification; Figure 5 This is a flowchart of another visual data processing method provided in one embodiment of this specification; Figure 6 This is a flowchart of the visual data processing method provided in one embodiment of this specification; Figure 7 This is a schematic diagram of the structure of a data processing system provided in one embodiment of this specification; Figure 8 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0015] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0016] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0017] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0018] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0019] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0020] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0021] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0022] VQA: Visual Question Answering, is a system that combines computer vision and natural language processing capabilities to answer free-form questions about image or video content.

[0023] Dialogue control center: The central hub responsible for responding to user voice interactions.

[0024] Intent understanding: In natural language processing, the intent of user language input is identified to facilitate subsequent classification and processing.

[0025] RTC: Real-Time Communication.

[0026] The GRTN (Global Real-time Transmission Network) is a global real-time transmission network designed to provide efficient, low-latency audio and video transmission services. Based on over 2,800 edge nodes worldwide, the GRTN network is a communication-grade transmission network. Through dynamic networking and proximity-based access, it enables users to access content from anywhere in the world, thereby reducing latency.

[0027] CDN node: A server node in a Content Delivery Network (CDN). CDN is a technology that improves content delivery speed and user experience by distributing content to nodes closest to the user. CDN nodes are typically distributed globally and can be physical servers, virtual servers, or cloud servers. By accessing the nearest CDN node, users can obtain the content they need faster, reducing latency and the impact of network congestion.

[0028] Video decoding is the crucial step in converting compressed video data into a visible image. When a video is recorded or created, the original image frames are compressed using a specific encoding algorithm to generate a smaller data stream suitable for storage and transmission. When the video needs to be viewed, the decoder uses the corresponding algorithm to parse this compressed data and reconstruct the original image frames.

[0029] VQA Agent: This refers to an intelligent agent used in visual question answering (VQA) tasks. It can complete visual question answering tasks by perceiving the environment, making decisions based on goals, and autonomously executing actions using tools.

[0030] OSS (Object Storage Service): refers to object storage service. OSS is a massive, secure, low-cost, and highly reliable cloud storage service suitable for storing files of any type.

[0031] Redis is an open-source in-memory database primarily used for storing key-value pairs. It supports various data types, including strings, hashes, lists, sets, and sorted sets.

[0032] UDP: is a protocol that operates at the transport layer in the OSI (Open Systems Interconnection) model.

[0033] AI PIN is a wearable device designed to reduce reliance on smartphones and offer a completely new way of interacting. Using a screenless interface and laser projection technology, AI PIN projects content onto the palm of the hand, providing a wealth of functionality and interactive experiences.

[0034] With the continuous development of artificial intelligence technology, neural network models can be applied to various data processing scenarios, such as processing visual data. However, in the process of processing visual data using neural network models, there is a problem of inaccurate data processing results.

[0035] For example, with the development of AI technology, more and more people are starting to use AI products. Among them, visual question answering, as an important branch of the AI ​​field, has been widely applied.

[0036] Traditional VQA mostly involves the user providing a static image and a question described in natural language. The VQA system then needs to understand the content of the image and the meaning of the question, and generate an accurate answer to the user.

[0037] In AI products with a first-person perspective, such as AI glasses, users not only want VQA functionality, but also hope that the AI ​​glasses can ask and answer questions based on a continuous video content, or guide them to complete a relatively complex task through continuous input of video stream. In this case, simply using a question-and-answer method would be very inefficient.

[0038] Currently, most VQA (Vacuum Question Answering) systems involve users providing a static image and a question described in natural language. The VQA system then needs to understand the content of the image and the meaning of the question to generate an accurate answer for the user. However, in the context of AI glasses, users not only need VQA but also expect the AI ​​glasses to be able to answer questions about a continuous video. In this case, simply using a question-and-answer approach would be extremely inefficient.

[0039] To address the aforementioned issues, this manual provides a solution for processing images captured by AI glasses; details can be found in [link to documentation]. Figure 1 , Figure 1 This is a schematic diagram of an image processing scheme provided in one embodiment of this specification; based on Figure 1 It is known that this solution uses voice interaction and traditional VQA voice / button photo taking methods for data interaction. Specifically, the solution uses the device's camera to take a photo, then uploads the image to the dialogue control center via HTTP. The user can then ask questions about the image content via voice. The user's voice is captured by microphone recording and uploaded to the dialogue control center via WebSocket. After processing, the dialogue control center performs intent understanding on the user's question and the image. If the intent understanding confirms a match with the Agent, the user's voice is converted into text and the image URL is sent to the VQA Agent for processing. The resulting response is then returned to the device, thus completing one round of dialogue. However, this solution has a problem: when users want to ask and answer questions about continuous video content, the traditional VQA method is relatively inefficient. On the one hand, after taking a photo, the image must be stored on the device's disk before it can be uploaded from the file system to the cloud's OSS storage space via HTTP. On the other hand, after obtaining the image URL, the cloud Agent also needs to read it from the OSS storage space before sending it to the large model for processing.

[0040] Based on this, a visual data processing method is provided in this specification. One or more embodiments of this specification also relate to another visual data processing method, a data processing apparatus, another data processing apparatus, a data processing system, an artificial intelligence wearable device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0041] See Figure 2 , Figure 2 A flowchart of a visual data processing method according to an embodiment of this specification is shown. The visual data processing method is applied to a visual data processing unit and specifically includes the following steps.

[0042] Step 202: Receive a data processing request for the target visual data, wherein the data processing request carries the target text associated with the target visual data, and the target text is obtained by text processing the target speech.

[0043] Step 204: Using a multimodal data processing model, perform visual processing on the target visual data based on the target text to obtain a first data processing result.

[0044] Step 206: Using a visual data processing model, perform visual processing on the target visual data to obtain a second data processing result, and obtain the target data processing result based on the first data processing result and the second data processing result.

[0045] Here, the target visual image can be understood as image data, video data, etc.; the target text can be understood as text used to process the target visual data, for example, the target text can be question text, task text, etc.; the data processing request can be understood as a request to process the target visual data, for example, the data processing request can be a visual question-and-answer request, a visual data analysis request, etc.; the target speech can be understood as speech data used to indicate what kind of processing operation to be performed on the target visual data, such as user speech.

[0046] The multimodal data processing model can be a model capable of processing multimodal data, and this multimodal data processing model can be a VL large model; the first data processing result can be the processing result corresponding to the target visual data; for example, visual question answers, user behavior guidance information, risk warning information, etc.; the visual data processing model can be a model that processes visual data (such as target visual data or visual data to be processed), and this visual data processing model can be a VL large model; the second data processing result can be the visual analysis result corresponding to the target visual data, such as visual question answers, user behavior guidance information, risk warning information, etc.

[0047] The target data processing result can be understood as the result obtained by fusing the first data processing result and the second data processing result.

[0048] The visual data processing unit can be understood as a unit that processes visual data. For example, the visual data processing unit can be a hardware device such as a server or computing node, or it can be a software device such as a cloud server, virtual machine, or application. In one or more embodiments of this specification, the visual data processing unit can be a continuous VQA Agent or a VQA Agent; in addition, there can be multiple visual data processing units.

[0049] The visual data processing unit can belong to a data processing system, which may include a visual data management unit and visual data processing units. The visual data management unit can be understood as a unit for managing operations such as the acquisition and transmission of visual data; the visual data management unit can also manage one or more visual data processing units, thereby controlling the visual data processing units to process the visual data; the visual data management unit can be a hardware device such as a server or computing node, or it can be a software device such as a cloud server, virtual machine, or application. In one or more embodiments of this specification, the visual data management unit can be a dialogue control center or a dialogue control module; furthermore, there can be only one visual data management unit. Figure 3 This is an execution flowchart of a data processing system provided in one embodiment of this specification, based on... Figure 3 It is known that the data processing system can perform data interaction using voice interaction and continuous VQA voice-activated photo taking / button-activated photo taking. Specifically, the device only needs to enter the camera's preview mode to take a picture, and then directly upload the acquired video stream to the GRTN in the cloud via the RTC channel. In addition, the device can capture audio streams by recording with a microphone, and then directly upload the acquired audio streams to the GRTN in the cloud via the RTC channel.

[0050] The central control unit pulls audio streams from the GRTN's CDN nodes, while the continuous VQA Agent pulls video streams from the GRTN's CDN nodes. The central control unit needs to perform intent recognition (i.e., intent understanding) on ​​the audio stream. If the intent matches the VQA Agent, it sends the start and end timestamps of the user's speech (i.e., the audio stream) and the content of the user's speech to the continuous VQA Agent. The continuous VQA Agent can then process the pulled video stream and / or the start and end timestamps and the content of the user's speech sent by the central control unit.

[0051] In one or more embodiments provided in this specification, the data processing request is a visual question-answering request; The step of using a multimodal data processing model to perform visual processing on the target visual data based on the target text to obtain a first data processing result includes: Using the multimodal data processing model, visual analysis is performed on the target visual data based on the target text to obtain the first data processing result and the result detection information corresponding to the first data processing result, wherein the result detection information includes result fusion information or result fusion information not performed; The step of using a visual data processing model to perform visual processing on the target visual data to obtain a second data processing result, and obtaining a target data processing result based on the first data processing result and the second data processing result, includes: When the result detection information is the execution result fusion information, the visual data processing model is used to perform visual processing on the target visual data to obtain the second data processing result; Based on the first data processing result and the second data processing result, the target data processing result corresponding to the visual question answering request is obtained.

[0052] Among these, result detection information can be understood as indicating whether it needs to be fused with the results of the visual data processing model. Performing result fusion information can be understood as indicating that it needs to be fused with the results of the visual data processing model. Not performing result fusion information can be understood as indicating that it does not need to be fused with the results of the visual data processing model.

[0053] Taking the application of the visual data processing method provided in this manual in a continuous VQA scenario as an example, this visual data processing method will be explained. The VL large model instance 2 (i.e., the multimodal data processing model) in the continuous VQA Agent (i.e., is also used in a separate process to respond to the user's voice query (i.e., the target voice). At this time, the image frame sequence (i.e., the target visual data) of the start and end time period of the user's voice query will be extracted and sent to the VL large model instance 2 for processing in combination with the user's voice input. The VL large model instance 2 will determine whether the result of this interaction can be directly returned to the user. If not, it will be handed over to the LLM decision model (i.e., the decision model) to perform the final result aggregation and obtain the aggregation result (i.e., the target data processing result). Thus, by aggregating the processing results of the two models, the accurate target data processing result is obtained.

[0054] In one or more embodiments provided in this specification, after performing visual analysis on the target visual data based on the target text using the multimodal data processing model to obtain the first data processing result and the result fusion detection information corresponding to the first data processing result, the method further includes: If the result detection information is the result fusion information not executed, the first data processing result is determined as the target data processing result corresponding to the visual question answering request.

[0055] Continuing with the previous example, the continuous VQA Agent retrieves the corresponding image frame sequence from the cache based on the corresponding timestamp, and sends it along with the user input to the VL large model instance 2 for processing. The VL large model instance 2 determines whether the interaction result can be directly returned to the user. If so, it feeds back the interaction result (i.e., the target data processing result) to the device (i.e., smart glasses, AI glasses, AI PIN). The interaction result is then displayed to the user through the device, achieving accurate and efficient user interaction.

[0056] In one or more embodiments provided in this specification, the target visual data is a plurality of target images; Before using a multimodal data processing model to perform visual processing on the target visual data based on the target text to obtain the first data processing result, the method further includes: Determine the target image identifiers corresponding to the plurality of target images carried in the data processing request; Based on the target image identifier, the plurality of target images are determined from the initial images stored in the data storage unit, wherein the initial images are obtained by image extraction from the initial video data, and the initial video data is video data collected by the data interaction unit using a visual data acquisition device.

[0057] The multiple target images can be a sequence of image frames in video data; the data storage unit can be understood as a unit used to store the initial image, such as memory, hard disk, database, or cloud storage service.

[0058] A data interaction unit can be understood as a unit capable of interacting with a user. For example, the data interaction unit can be smart glasses, AI glasses, terminal devices, etc. In one or more embodiments provided in this specification, the data interaction unit can be an artificial intelligence wearable device; the artificial intelligence wearable device can be understood as a wearable device based on artificial intelligence, for example, the artificial intelligence wearable device can be smart glasses, AI glasses, AI PIN, etc.; the artificial intelligence wearable device can achieve better functionality by interacting with the cloud (e.g., a data processing system, a visual data management unit, a visual data processing unit).

[0059] A target image identifier can be understood as information used to identify multiple target images. For example, the target image identifier can be the name of the target image, the timestamp of the target image, the start time of multiple target images, etc.

[0060] Using the previous example, the AI ​​glasses device (i.e., the data interaction unit) can turn on the camera to capture the scene and push the scene to the nearest CDN node in the GRTN network in real time; the continuous VQA Agent (i.e., the visual data processing unit) pulls the video stream (i.e., video data) from the CDN node; then it decodes and extracts frames from the video stream to obtain a sequence of image frames (i.e., the initial image), where the frequency of frame extraction is configurable.

[0061] The extracted images are stored sequentially in OSS, and the timestamp and OSS storage path are recorded in Redis. When the VQA Agent receives the start and end timestamps (i.e., target image identifiers) and the content of the user's speech (target text) sent by the dialog control center, it retrieves the corresponding image frame sequence (i.e., multiple target images) from the cache based on the corresponding start and end timestamps (i.e., target image identifiers).

[0062] Based on the above embodiments, this method employs RTC technology, where all image data is stored in memory. The device (AI glasses) only needs to enter the camera's preview mode and then directly upload the acquired images from memory to the cloud via the RTC channel. The RTC protocol supports simultaneous transmission of audio and video streams, reducing the overhead of multiple image storage and retrieval operations. Furthermore, the underlying RTC protocol uses UDP for transmission, resulting in a faster transmission rate than TCP at the lower protocol level. Simultaneously, relying on the global network nodes of GRTN, the transmission speed of real-time audio and video (RTC) is even faster. This achieves low latency and improves transmission efficiency by employing RTC technology on the audio and video links, reducing disk I / O operations.

[0063] In one or more embodiments provided in this specification, the data interaction unit is smart glasses; After determining the plurality of target images from the initial images stored in the data storage unit based on the target image identifier, the method further includes: The system receives the initial video data sent by the smart glasses via a data transmission link. The initial video data is stored in the memory of the smart glasses and obtained by performing real-time video data acquisition operations through a visual data acquisition device configured on the smart glasses. The data transmission link is a link composed of multiple data transmission nodes used for data transmission between the smart glasses and the visual data processing unit. After obtaining the target data processing result based on the first data processing result and the second data processing result, the method further includes: The target data processing results are sent to the smart glasses for display.

[0064] The data transmission link can be understood as an RTC communication link between the data interaction unit and the visual data processing unit, or an Internet of Things (IoT) communication channel, etc.

[0065] A visual data acquisition device can be understood as a device used to acquire visual data of a target. This visual data acquisition device can be an image acquisition device or a video acquisition device; for example, the visual data acquisition device can be a camera, webcam, imager, etc.

[0066] Real-time video data acquisition can be understood as the operation of a visual data acquisition device to acquire target visual data in real time, such as the image acquisition operation performed by a camera in preview mode.

[0067] A data transmission node can be understood as a node in a data transmission link used for data transmission. For example, a data transmission node can be a router, CDN node, edge node, server, etc., without specific restrictions.

[0068] Following the previous example, the AI ​​glasses device (i.e., smart glasses) in this method can use RTC technology for data transmission. All image data in the process is stored in memory. The device (i.e., smart glasses) only needs to enter the preview mode of the camera (i.e., visual data acquisition device). The images acquired in the preview mode can be stored in memory, and then the acquired images can be directly uploaded to the cloud (e.g., data processing system, visual data management unit, or visual data processing unit) through the RTC channel (i.e., data transmission channel) in memory.

[0069] Specifically, the AI ​​glasses device can turn on the camera to capture the scene and push the scene stored in the memory to the nearest CDN node (i.e., data transmission node) in the GRTN network in real time; the continuous VQA Agent (i.e., visual data processing unit) pulls the video stream (i.e., video data) from the CDN node.

[0070] Once the interaction result is determined, it can be fed back to the device. The device then displays the result (i.e., the target data processing result) to the user. This enables continuous VQA to perform question-and-answer sessions on video streams, improving the user experience in hardware categories such as AI glasses.

[0071] In one or more embodiments provided in this specification, the method further includes: Receive a visual analysis request for visual data to be processed, wherein the visual analysis request carries a data identifier and visual data processing parameters of the visual data to be processed; The visual data to be processed is determined based on the data identifier, and the visual data processing model and decision module are used to perform visual processing on the visual data to be processed and the visual data processing parameters to obtain visual analysis results.

[0072] Visual analysis requests can be understood as requests to analyze and process visual data, such as image content recognition requests.

[0073] Visual data processing parameters can be understood as parameters used to process visual data. For example, these parameters can be system prompts and user-defined task objectives.

[0074] Visual analysis results can be understood as the results obtained from analyzing the visual data to be processed. For example, visual analysis results can be interactive results such as answers to visual questions, risk warning information, and user behavior guidance information.

[0075] Continuing with the previous example, the continuous VQA Agent (i.e., the visual data processing unit) pulls the video stream (i.e., video data) from the CDN node. Next, it decodes and extracts frames from the video stream to obtain a sequence of image frames (i.e., the initial images). Then, using the LLM decision model and VL large model instance 1, it processes these images, system prompts, and the user's initially set task objectives (task objective information) to obtain the final interaction result (i.e., the visual analysis result). The system prompts and the user's initially set task objective information can be sent by the dialogue control module (i.e., the visual data management unit).

[0076] In one or more embodiments provided in this specification, the visual data to be processed is a plurality of images to be processed; The process of determining the visual data to be processed based on the data identifier, and using the visual data processing model and decision module to perform visual processing on the visual data to obtain visual analysis results includes: Based on the data identifier, the plurality of images to be processed are determined from the initial images in the data storage unit, wherein the initial images are obtained by image extraction from the initial video data, and the initial video data is video data collected by the data interaction unit using a visual data acquisition device; Using the aforementioned visual data processing model, visual processing is performed on each image to be processed to obtain the third data processing result corresponding to each image to be processed. By using a decision model, multiple third-party data processing results are fused to obtain the visual analysis results. After using the visual data processing model and decision module to perform visual processing on the visual data to be processed and obtain visual analysis results, the process further includes: The visual analysis results are sent to the data interaction unit for display.

[0077] The data identifier can be understood as information used to identify multiple images to be processed. For example, the data identifier can be the name of the image to be processed, the timestamp of the image to be processed, the start time of multiple images to be processed, etc.

[0078] The third data processing result can be understood as the processing result corresponding to the image to be processed; for example, image analysis results, image content recognition results, user behavior guidance information, risk warning information, etc.

[0079] Following the previous example, the extracted images are stored sequentially in OSS, and the timestamp and OSS storage path are recorded in Redis. These images, along with the system prompt and the user's initial task objective, are then sent to VL large model instance 1 for processing to obtain the output results of VL large model instance 1 for each image frame.

[0080] In VL Large Model Example 1, the output of each frame of the image is input into the LLM decision model. The decision model performs the final content integration to obtain the final interaction result, which is then returned to the user. This process does not rely on the user's voice prompts. The LLM decision model will decide whether to actively prompt the user for the next step or correct current errors based on the goal and the current video content.

[0081] Figure 4 This is a schematic diagram of the interactive process of a visual data processing method provided in one embodiment of this specification; based on Figure 4As can be seen, the visual data processing method provided in this manual can automatically send responses and supports a question-and-answer format. Based on traditional natural language and image processing, this method changes the user input from a single static image to a real-time video stream. The cloud continuously feeds consecutive image frames into a large model for processing through frame extraction, allowing the large model to perform continuous VQA (Visual Quality Assurance). It can even proactively provide feedback based on the video content when the user does not ask a question (i.e., no voice input). For example, in a teaching process, the interaction between this method and the user is as follows: When the teaching mode is based on the user's voice, the model processes the video stream and provides a model response; even if the user does not ask a question (i.e., no voice input), it will proactively provide feedback. When the user proactively asks a question (i.e., a voice inquiry), the model processes the question and provides feedback, thus achieving automatic response sending and supporting a question-and-answer format. Finally, when the user exits the teaching mode based on their voice, the model provides an exit response, thus ending the interaction.

[0082] The visual data processing method applied to a visual data processing unit in one or more embodiments of this specification, upon receiving a data processing request for target visual data, utilizes a multimodal data processing model to perform visual processing on the target visual data based on the target text carried in the data processing request, obtaining a first data processing result; and then utilizes a visual data processing model to perform visual processing on the target visual data, obtaining a second data processing result; finally, through the first and second data processing results obtained from the multimodal data processing model and the visual data processing model, an accurate target data processing result is obtained, thus avoiding the problem of inaccurate data processing results.

[0083] See Figure 5 , Figure 5 A flowchart of another visual data processing method according to an embodiment of this specification is shown. This visual data processing method is applied to a visual data management unit and specifically includes the following steps.

[0084] Step 502: Perform speech processing on the target speech to obtain the speech processing result of the target speech, wherein the target speech is the speech associated with the target visual data.

[0085] Step 504: If the target speech meets the model processing conditions based on the speech processing results, a data processing request is determined based on the target text corresponding to the target speech, wherein the target text is obtained by performing text processing on the target speech.

[0086] Step 506: Send the data processing request to the visual data processing unit so that the visual data processing unit executes the steps of the above-described visual data processing method.

[0087] In one or more embodiments provided in this specification, the step of performing speech processing on the target speech to obtain the speech processing result of the target speech includes: The target speech is processed to obtain the target text; The target text is semantically analyzed using a text analysis module to obtain the language processing results of the target speech.

[0088] The text analysis module can be understood as a model or program module that performs semantic analysis on the target text. For example, the text analysis module can be an intent recognition model, a large model, a deep learning model, etc.

[0089] The language processing result can be understood as the semantic analysis result of the target text; the language processing result can be data indicating whether visual data processing unit needs to be used for visual data processing; for example, the language processing result can be 1 or 0, where 1 indicates that visual data processing unit needs to be used for visual data processing, and 0 indicates that visual data processing unit does not need to be used for visual data processing.

[0090] Using the previous example, the dialogue controller can pull the audio stream (i.e., the target speech) from the CDN node. Specifically, when the device detects that the user has started speaking, it can upload the audio stream to the CDN node, and the dialogue controller can pull the audio stream from the CDN node. Then, the dialogue controller converts the speech into text, which is then judged by the intent recognition module (i.e., the text analysis module). If it matches the VQA Agent (i.e., the language processing result of the intent recognition module indicates that it needs to be processed by the VQA Agent), it will send the start and end times of the user's speech and the content of the user's speech to the continuous VQA Agent.

[0091] Subsequently, the VQA Agent retrieves the corresponding image frame sequence from the cache based on the response timestamp (i.e., start and end time points), and sends it along with the user input to the VL large model instance 2 for processing. The VL large model instance 2 will determine whether the result of this interaction can be directly returned to the user. If not, it will be handed over to the LLM decision model for final result aggregation.

[0092] In one or more embodiments provided in this specification, before performing speech processing on the target speech to obtain the speech processing result of the target speech, the method further includes: The system receives a data interaction request sent by a data interaction unit and establishes a data transmission link between the data interaction unit and the visual data management unit based on the data interaction request. The data transmission link is a link composed of multiple data transmission nodes used for data transmission between the data interaction unit and the visual data management unit. The data interaction unit receives the target voice sent by the data transmission link, wherein the target voice is stored in the memory of the data interaction unit and obtained by performing real-time voice acquisition operation through a voice acquisition device configured on the data interaction unit.

[0093] The data transmission link can be understood as an RTC communication link between the data interaction unit and the visual data management unit, or an Internet of Things (IoT) communication channel, etc.

[0094] A voice acquisition device can be understood as a device used to acquire target voice, such as a microphone, a recorder, etc.

[0095] Continuing with the previous example, the user can initiate a video conversation request (i.e., a data interaction request) via voice or button. Upon receiving the request, the central control unit establishes an RTC communication link (i.e., a data transmission link) and notifies the device. The device then activates its microphone to capture the user's voice (i.e., the target voice) and pushes the user's voice stored in memory to the nearest CDN node in the GRTN network in real time. The central control unit can then pull the audio stream (i.e., the target voice) from the CDN node. Because this method uses RTC technology, the RTC protocol supports the simultaneous transmission of audio and video streams, reducing the overhead of multiple storage and retrieval of images and audio. Furthermore, relying on the global network nodes of GRTN, the transmission speed of real-time audio and video (RTC) is faster.

[0096] The visual data processing method applied to a visual data management unit in one or more embodiments of this specification, based on the speech processing results of the target speech and determining that the target speech meets the model processing conditions, determines a data processing request based on the target text corresponding to the target speech, and sends the data processing request to the visual data processing unit. Upon receiving the data processing request for the target visual data, the visual data processing unit utilizes a multimodal data processing model to perform visual processing on the target visual data according to the target text carried in the data processing request, obtaining a first data processing result; and then utilizes a visual data processing model to perform visual processing on the target visual data, obtaining a second data processing result; finally, through the first and second data processing results obtained from the multimodal data processing model and the visual data processing model, an accurate target data processing result is obtained, avoiding the problem of inaccurate data processing results.

[0097] The following is in conjunction with the appendix Figure 6 Taking the application of the visual data processing method provided in this specification in a continuous VQA scenario as an example, the visual data processing method will be further explained. Figure 6 A flowchart illustrating the processing procedure of a visual data processing method provided in one embodiment of this specification is shown.

[0098] based on Figure 6 As can be seen, this method employs three large model instances for parallel processing to improve the real-time response across the entire process on the algorithm side; these three large model instances are: two VL vision large models and one LLM decision model. For example... Figure 4 As shown in the flowchart, VL model instance 1 continuously processes the image sequence input from the streaming end within a separate process. Simultaneously, it inputs the user-defined task objective (if any) and system prompts into VL model instance 1. At this point, each image frame generates a corresponding output. These intermediate results are then fed to the LLM decision model, which determines whether to proactively push feedback to the user based on the user-defined task objective and the processing results of each image frame.

[0099] Based on this, the method specifically includes the following steps.

[0100] Step 602: On the device side, upload data.

[0101] 1. Users can initiate video conversation requests to the dialogue control module via voice or by pressing buttons on the device; 2. The dialogue control module, upon receiving a video dialogue request, establishes an RTC communication link with the device and notifies the device (i.e., the AI ​​glasses).

[0102] 3. On the device side, turn on the camera to capture the image and push the image in real time to the nearest CDN node in the GRTN network.

[0103] 4. On the device side, when the user starts speaking, the audio stream is uploaded to the nearest CDN node in the GRTN network.

[0104] Step 604: The central control module checks whether the device has uploaded an audio stream. If yes, proceed to step 606; otherwise, proceed to step 614.

[0105] Step 606: The continuous VQA Agent pulls the video stream from the CDN node according to the control instructions sent by the dialogue control module, and performs decoding and frame extraction processing on the video stream to obtain the image frame sequence. The frame extraction frequency is configurable.

[0106] It should be noted that the dialogue control module can be understood as a central scheduling module, while VQA is more like one or more executors; The dialogue control module can send control commands to the VQA Agent, which can include system prompts and user-defined task objectives. Under the instructions of the dialogue control module, the VQA Agent pulls video streams from CDN nodes and processes the image frame sequences in the video streams.

[0107] Afterwards, the VQA Agent can continuously feed consecutive image frames into the large model (VL large model instance 1, LLM decision model) for processing, and use a multi-instance collaborative approach on the algorithm side to reduce the latency of the entire link. In this way, the large model can perform continuous VQA, and can even proactively provide feedback based on the video content without the user asking questions, to guide and correct the user's behavior, thereby completing the task objective.

[0108] Step 608: The VQA Agent continuously stores the image frame sequence obtained by frame extraction into OSS in order, and records the timestamp and OSS storage path in Redis.

[0109] Step 610: The VQA Agent continuously sends these image frames, system prompts, and user-defined task objectives to the VL large model instance 1 for processing.

[0110] It should be noted that, since the VQA Agent is a passive responder, the dialogue control module adjusts the system prompts based on the user's current voice input. In actual use, the intent recognition module can identify different user intents, thereby determining the task objective.

[0111] For example, a user might ask the glasses to guide them in playing chess, cooking, or other tasks—scenarios with clear objectives that will continue for multiple rounds. In these cases, the dialogue control module will use one set of prompts. However, if the user simply wants the glasses to translate or take a picture of a plant, it's a single-round chat, requiring a different set of prompts.

[0112] Furthermore, when a user needs to change the task objective, they will input the voice message for changing the task objective into the dialogue control module through the device. The dialogue control module will use an intent recognition model to analyze the intent of the user's voice message for changing the task objective, and modify the historical task objective to obtain the updated task objective.

[0113] Step 612: The VQA Agent uses VL pattern instance 1 to perform intent analysis on each frame of the image and generates corresponding outputs (i.e., intent analysis results). These intermediate outputs are then fed into the LLM decision model for integration to obtain the interaction results (i.e., interaction information) fed back to the user.

[0114] Specifically, the VQA Agent uses VL mode instance 1 to perform intent analysis on each frame of the image and generates the corresponding output (i.e., intent analysis results). These intermediate outputs are fed into the LLM decision model, which then decides whether to provide new feedback to the user based on the user's task objectives and the results of processing each frame of the image. Finally, the decision model integrates the content and returns it to the user.

[0115] It should be noted that this process does not rely on the user's voice prompts. The LLM decision model will decide whether to actively prompt the user for the next step or correct the current error based on the goal and the current video content.

[0116] Step 614: The dialogue control module converts the voice (i.e., audio stream) obtained from the CDN node into speech text, and uses the intent recognition model to perform semantic analysis on the speech text to determine whether to use the VQA Agent for processing.

[0117] If a VQA Agent is hit, the start and end times of the user's speech, along with the audio text (i.e., the content of the user's speech), will be sent to the next VQA Agent.

[0118] If no VQA Agent is matched, the dialog controller will distribute the request to other agents for processing, such as the device control agent.

[0119] Step 616: The continuous VQA Agent receives the start and end timestamps and voice text sent by the dialogue control module; and retrieves the corresponding image frame sequence from the cache according to the response timestamps (start and end timestamps).

[0120] Step 618: The continuous VQA Agent sends the image frame sequence and the user-input voice content together to the VL large model instance 2 for processing to obtain the interaction result.

[0121] Step 620: Continuously use the VQA Agent to determine whether the interaction result can be fed back to the user.

[0122] Specifically, VL large model instance 2 will determine whether the result of this interaction can be directly returned to the user.

[0123] If the result cannot be returned to the user, the result of this interaction is sent to the LLM decision model. The LLM decision model will merge the intent analysis result output by VL large model instance 1 after processing the image frame with the result of this interaction, thereby obtaining the interaction information to be fed back to the user by merging the processing results of VL large model instance 1 and VL large model instance 2.

[0124] Step 622: Feed back the interaction result to the device; display the interaction result to the user through the device.

[0125] It should be noted that in the visual data processing method provided in this manual, after the dialogue control module converts the user's input speech into speech text, it will detect whether the speech text has any text violations. If the voice text contains a text violation, a specific message is sent to the device (e.g., "What you said contains a violation. Please explain your needs again").

[0126] If the speech text does not contain any textual violations, then the intention recognition model is used to perform semantic recognition processing on the speech text.

[0127] Furthermore, when the VQA Agent obtains image frames from the video stream, it will detect whether the image frame has any violations. If the image frame contains a violation, a specific message is sent to the device (e.g., "The content of the image you captured contains a violation. Please re-capture the video data").

[0128] If the image frame does not contain any image violations, the extracted images will be stored in OSS in sequence.

[0129] Based on the above embodiments, it can be seen that the visual data processing method in this specification provides a continuous VQA system based on a large visual model. The system adopts RTC technology, and all image data in the process is in memory. The device (i.e., AI glasses) only needs to enter the camera's preview mode, and then directly upload the acquired image to the cloud through the RTC channel in memory. The RTC protocol supports the simultaneous transmission of audio streams and video streams, reducing the overhead of multiple image storage and retrieval.

[0130] Furthermore, the underlying RTC protocol uses UDP for transmission, resulting in a faster transmission rate at the lower protocol level compared to TCP. Simultaneously, thanks to the global network node support of GRTN, real-time audio and video (RTC) transmission speeds are even faster.

[0131] Furthermore, based on traditional natural language and image input, this system changes the user input from a single static image to a real-time video stream. The cloud continuously feeds consecutive image frames into a large model for processing through frame extraction. At the algorithm level, a multi-instance collaborative approach is used to reduce latency across the entire chain. In this way, the large model can perform continuous VQA and even proactively provide feedback based on the video content without the user asking any questions, guiding and correcting the user's behavior to achieve the task objectives.

[0132] Based on the above, this method achieves continuous VQA, enabling question-and-answer processing of video streams and improving user experience in hardware products such as AI glasses. Furthermore, this method achieves low latency by employing RTC technology on the audio and video link to improve transmission efficiency and reduce disk I / O operations. On the algorithm side, multi-instance collaborative processing is used.

[0133] Corresponding to the above method embodiments, this specification also provides data processing system embodiments. Figure 7 A schematic diagram of the structure of a data processing system according to one embodiment of this specification is shown. Figure 7 As shown, the system includes a visual data management unit 702 and a visual data processing unit 704; The visual data management unit 702 is configured to perform speech processing on the target speech to obtain the speech processing result of the target speech, wherein the target speech is speech associated with the target visual data. When it is determined that the target speech meets the model processing conditions based on the speech processing result, a data processing request is determined based on the target text corresponding to the target speech, wherein the target text is obtained by performing text processing on the target speech. The data processing request is then sent to the visual data processing unit 704. The visual data processing unit 704 is configured to receive a data processing request for the target visual data, wherein the data processing request carries the target text associated with the target visual data, performs visual processing on the target visual data based on the target text using a multimodal data processing model to obtain a first data processing result, performs visual processing on the target visual data using the visual data processing model to obtain a second data processing result, and obtains a target data processing result based on the first data processing result and the second data processing result.

[0134] The data processing system described in one or more embodiments of this specification utilizes a visual data management unit to generate a data processing request and sends the request to the visual data processing unit. Upon receiving a data processing request for target visual data, the visual data processing unit uses a multimodal data processing model to perform visual processing on the target visual data based on the target text carried in the request, obtaining a first data processing result. It then uses the same visual data processing model to perform further visual processing on the target visual data, obtaining a second data processing result. Finally, by combining the first and second data processing results obtained from the multimodal data processing model and the visual data processing model, an accurate target data processing result is obtained, thus avoiding the problem of inaccurate data processing results.

[0135] The above is an illustrative scheme of a data processing system according to this embodiment. It should be noted that the technical solution of this data processing system belongs to the same concept as the technical solution of any of the above-described visual data processing methods. For details not described in detail in the technical solution of the data processing system, please refer to the description of the technical solution of any of the above-described visual data processing methods.

[0136] Corresponding to the above method embodiments, this specification also provides an embodiment of an artificial intelligence wearable device, which is configured as follows: Real-time visual data is acquired according to the target's perspective to obtain target visual data, and real-time voice is acquired to obtain target voice. The target's perspective is the same as the human eye's first-person perspective. The target visual data and the target speech are sent to the data processing system, and the target data processing result sent by the data processing system is received, wherein the data processing system is the data processing system in one or more of the above embodiments.

[0137] The wearable AI device is equipped with a visual data acquisition device, which can collect real-time visual data from the target's perspective to obtain the target's visual data. It also has a voice acquisition device, which can collect real-time voice data to obtain the target's speech. This target's visual perspective can serve as the wearable AI device's first-person viewpoint.

[0138] The above is an illustrative scheme of an artificial intelligence wearable device according to this embodiment. It should be noted that the technical solution of this artificial intelligence wearable device belongs to the same concept as any of the above-mentioned visual data processing methods and data processing systems. For details not described in detail in the technical solution of the artificial intelligence wearable device, please refer to the description of any of the above-mentioned visual data processing methods and data processing systems.

[0139] Corresponding to the above method embodiments, this specification also provides an embodiment of a visual data processing device, which is applied to a visual data processing unit and includes: The request receiving module is configured to receive a data processing request for target visual data, wherein the data processing request carries target text associated with the target visual data, and the target text is obtained by text processing the target speech. The first data processing module is configured to use a multimodal data processing model to perform visual processing on the target visual data based on the target text to obtain a first data processing result. The second data processing module is configured to use a visual data processing model to perform visual processing on the target visual data to obtain a second data processing result, and to obtain a target data processing result based on the first data processing result and the second data processing result.

[0140] Optionally, the data processing request is a visual question-answering request; The first data processing module is further configured as follows: Using the multimodal data processing model, visual analysis is performed on the target visual data based on the target text to obtain the first data processing result and the result detection information corresponding to the first data processing result, wherein the result detection information includes result fusion information or result fusion information not performed; The second data processing module is also configured as follows: When the result detection information is the execution result fusion information, the visual data processing model is used to perform visual processing on the target visual data to obtain the second data processing result; Based on the first data processing result and the second data processing result, the target data processing result corresponding to the visual question answering request is obtained.

[0141] Optionally, the visual data processing device further includes a result determination module, configured to: If the result detection information is the result fusion information not executed, the first data processing result is determined as the target data processing result corresponding to the visual question answering request.

[0142] Optionally, the target visual data may be multiple target images; The visual data processing device further includes a data acquisition module, configured as follows: Determine the target image identifiers corresponding to the plurality of target images carried in the data processing request; Based on the target image identifier, the plurality of target images are determined from the initial images stored in the data storage unit, wherein the initial images are obtained by image extraction from the initial video data, and the initial video data is video data collected by the data interaction unit using a visual data acquisition device.

[0143] Optionally, the data interaction unit is smart glasses; The data acquisition module is further configured to: The initial video data sent by the smart glasses is acquired, and image extraction is performed on the initial video data to obtain the initial image; The visual data processing device further includes a result sending module, configured as follows: The target data processing results are sent to the smart glasses for display.

[0144] Optionally, the visual data processing device further includes a third data processing module, configured to: Receive a visual analysis request for visual data to be processed, wherein the visual analysis request carries a data identifier and visual data processing parameters of the visual data to be processed; The visual data to be processed is determined based on the data identifier, and the visual data processing model and decision module are used to perform visual processing on the visual data to be processed and the visual data processing parameters to obtain visual analysis results.

[0145] Optionally, the visual data to be processed consists of multiple images to be processed; The third data processing module is further configured to: Based on the data identifier, the plurality of images to be processed are determined from the initial images in the data storage unit, wherein the initial images are obtained by image extraction from the initial video data, and the initial video data is video data collected by the data interaction unit using a visual data acquisition device; Using the aforementioned visual data processing model, visual processing is performed on each image to be processed to obtain the third data processing result corresponding to each image to be processed. By using a decision model, multiple third-party data processing results are fused to obtain the visual analysis results. The visual data processing device further includes a result sending module, configured as follows: The visual analysis results are sent to the data interaction unit for display.

[0146] In one or more embodiments of this specification, a visual data processing device applied to a visual data processing unit, upon receiving a data processing request for target visual data, utilizes a multimodal data processing model to perform visual processing on the target visual data based on the target text carried in the data processing request, obtaining a first data processing result; and then utilizes a visual data processing model to perform visual processing on the target visual data, obtaining a second data processing result; finally, through the first and second data processing results obtained from the multimodal data processing model and the visual data processing model, an accurate target data processing result is obtained, thus avoiding the problem of inaccurate data processing results.

[0147] The above is an illustrative scheme of a visual data processing device according to this embodiment. It should be noted that the technical solution of this visual data processing device and the technical solution of the visual data processing method described above belong to the same concept. For details not described in detail in the technical solution of the visual data processing device, please refer to the description of the technical solution of the visual data processing method described above.

[0148] Corresponding to the above method embodiments, this specification also provides another embodiment of a visual data processing device, which is applied to a visual data management unit, and the device includes: The speech processing module is configured to perform speech processing on the target speech to obtain the speech processing result of the target speech, wherein the target speech is speech associated with the target visual data; The request determination module is configured to determine a data processing request based on the target text corresponding to the target speech, provided that the target speech meets the model processing conditions based on the speech processing results; wherein the target text is obtained by performing text processing on the target speech. The request sending module is configured to send the data processing request to the visual data processing unit so that the visual data processing unit performs the steps of the above-described visual data processing method.

[0149] Optionally, the voice processing module is further configured to: The target speech is processed to obtain the target text; The target text is semantically analyzed using a text analysis module to obtain the language processing results of the target speech.

[0150] Optionally, the visual data processing device further includes a data acquisition module, configured to: The system receives a data interaction request sent by a data interaction unit, and based on the data interaction request, acquires the target speech sent by the data interaction unit, wherein the target speech is the speech acquired by the data interaction unit using a speech data acquisition device.

[0151] The visual data processing device applied to the visual data management unit in one or more embodiments of this specification, based on the speech processing results of the target speech and determining that the target speech meets the model processing conditions, determines a data processing request based on the target text corresponding to the target speech, and sends the data processing request to the visual data processing unit. Upon receiving the data processing request for the target visual data, the visual data processing unit utilizes a multimodal data processing model to perform visual processing on the target visual data according to the target text carried in the data processing request, obtaining a first data processing result; and then utilizes the visual data processing model to perform visual processing on the target visual data, obtaining a second data processing result; finally, through the first and second data processing results obtained from the multimodal data processing model and the visual data processing model, an accurate target data processing result is obtained, avoiding the problem of inaccurate data processing results.

[0152] The above is an illustrative scheme of another visual data processing device according to this embodiment. It should be noted that the technical solution of this other visual data processing device and the technical solution of the other visual data processing method described above belong to the same concept. For details not described in detail in the technical solution of the other visual data processing device, please refer to the description of the technical solution of the other visual data processing method described above.

[0153] Figure 8 A structural block diagram of a computing device 800 according to one embodiment of this specification is shown. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0154] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0155] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0156] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 800 can also be a mobile or stationary server.

[0157] The processor 820 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of any of the above methods.

[0158] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to any of the above-described method embodiments, so the description is relatively simple; relevant parts can be referred to in the description of any of the above-described method embodiments.

[0159] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0160] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to any of the above-described method embodiments, so the description is relatively simple; relevant parts can be referred to in the description of any of the above-described method embodiments.

[0161] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0162] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solution of any of the above methods, and any details not described in detail in the technical solution of the computer program product can be referred to the description of the technical solution of any of the above methods.

[0163] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0164] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0165] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0166] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0167] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A visual data processing method, applied to a visual data processing unit, comprising: Receive a data processing request for target visual data, wherein the data processing request carries target text associated with the target visual data, and the target text is obtained by text processing the target speech; Using a multimodal data processing model, visual processing is performed on the target visual data based on the target text to obtain a first data processing result; The target visual data is processed using a visual data processing model to obtain a second data processing result, and the target data processing result is obtained based on the first data processing result and the second data processing result.

2. The visual data processing method according to claim 1, wherein the data processing request is a visual question-answering request; The step of using a multimodal data processing model to perform visual processing on the target visual data based on the target text to obtain a first data processing result includes: Using the multimodal data processing model, visual analysis is performed on the target visual data based on the target text to obtain the first data processing result and the result detection information corresponding to the first data processing result, wherein the result detection information includes result fusion information or result fusion information not performed; The step of using a visual data processing model to perform visual processing on the target visual data to obtain a second data processing result, and obtaining a target data processing result based on the first data processing result and the second data processing result, includes: When the result detection information is the execution result fusion information, the visual data processing model is used to perform visual processing on the target visual data to obtain the second data processing result; Based on the first data processing result and the second data processing result, the target data processing result corresponding to the visual question answering request is obtained.

3. The visual data processing method according to claim 2, further comprising, after using the multimodal data processing model to perform visual analysis on the target visual data based on the target text to obtain the first data processing result and the result fusion detection information corresponding to the first data processing result, the method further comprises: If the result detection information is the result fusion information not executed, the first data processing result is determined as the target data processing result corresponding to the visual question answering request.

4. The visual data processing method according to any one of claims 1 to 3, wherein the target visual data comprises multiple target images; Before using a multimodal data processing model to perform visual processing on the target visual data based on the target text to obtain the first data processing result, the method further includes: Determine the target image identifiers corresponding to the plurality of target images carried in the data processing request; Based on the target image identifier, the plurality of target images are determined from the initial images stored in the data storage unit, wherein the initial images are obtained by image extraction from the initial video data, and the initial video data is video data collected by the data interaction unit using a visual data acquisition device.

5. The visual data processing method according to claim 4, wherein the data interaction unit is smart glasses; After determining the plurality of target images from the initial images stored in the data storage unit based on the target image identifier, the method further includes: The system receives the initial video data sent by the smart glasses via a data transmission link. The initial video data is stored in the memory of the smart glasses and obtained by performing real-time video data acquisition operations through a visual data acquisition device configured on the smart glasses. The data transmission link is a link composed of multiple data transmission nodes used for data transmission between the smart glasses and the visual data processing unit. Image extraction is performed on the initial video data to obtain the initial image; After obtaining the target data processing result based on the first data processing result and the second data processing result, the method further includes: The target data processing results are sent to the smart glasses for display.

6. The visual data processing method according to claim 1, further comprising: Receive a visual analysis request for visual data to be processed, wherein the visual analysis request carries a data identifier and visual data processing parameters of the visual data to be processed; The visual data to be processed is determined based on the data identifier, and the visual data processing model and decision module are used to perform visual processing on the visual data to be processed and the visual data processing parameters to obtain visual analysis results.

7. The visual data processing method according to claim 6, wherein the visual data to be processed comprises multiple images to be processed; The process of determining the visual data to be processed based on the data identifier, and using the visual data processing model and decision module to perform visual processing on the visual data to obtain visual analysis results includes: Based on the data identifier, the plurality of images to be processed are determined from the initial images in the data storage unit, wherein the initial images are obtained by image extraction from the initial video data, and the initial video data is video data collected by the data interaction unit using a visual data acquisition device; Using the aforementioned visual data processing model, visual processing is performed on each image to be processed to obtain the third data processing result corresponding to each image to be processed. By using a decision model, multiple third-party data processing results are fused to obtain the visual analysis results. After using the visual data processing model and decision module to perform visual processing on the visual data to be processed and obtain visual analysis results, the process further includes: The visual analysis results are sent to the data interaction unit for display.

8. A visual data processing method, applied to a visual data management unit, the method comprising: The target speech is subjected to speech processing to obtain the speech processing result of the target speech, wherein the target speech is speech associated with the target visual data; If, based on the speech processing results, it is determined that the target speech meets the model processing conditions, a data processing request is determined based on the target text corresponding to the target speech, wherein the target text is obtained by performing text processing on the target speech; The data processing request is sent to the visual data processing unit so that the visual data processing unit performs the steps of the visual data processing method in claim 1.

9. The visual data processing method according to claim 8, wherein performing speech processing on the target speech to obtain the speech processing result of the target speech includes: The target speech is processed to obtain the target text; The target text is semantically analyzed using a text analysis module to obtain the language processing results of the target speech.

10. The visual data processing method according to claim 8, further comprising, before performing speech processing on the target speech to obtain the speech processing result of the target speech: The system receives a data interaction request sent by a data interaction unit and establishes a data transmission link between the data interaction unit and the visual data management unit based on the data interaction request. The data transmission link is a link composed of multiple data transmission nodes used for data transmission between the data interaction unit and the visual data management unit. The data interaction unit receives the target voice sent by the data transmission link, wherein the target voice is stored in the memory of the data interaction unit and obtained by performing real-time voice acquisition operation through a voice acquisition device configured on the data interaction unit.

11. A data processing system, the data processing system comprising a visual data management unit and a visual data processing unit; The visual data management unit is configured to perform speech processing on the target speech to obtain the speech processing result of the target speech, wherein... The target speech is speech associated with the target visual data. If the target speech meets the model processing conditions based on the speech processing results, a data processing request is determined based on the target text corresponding to the target speech. The target text is obtained by performing text processing on the target speech. The data processing request is then sent to the visual data processing unit. The visual data processing unit is configured to receive a data processing request for the target visual data, wherein the data processing request carries the target text associated with the target visual data, performs visual processing on the target visual data based on the target text using a multimodal data processing model to obtain a first data processing result, performs visual processing on the target visual data using the visual data processing model to obtain a second data processing result, and obtains a target data processing result based on the first data processing result and the second data processing result.

12. An artificial intelligence wearable device, configured as follows: Real-time visual data is acquired from the target's perspective to obtain target visual data, and real-time voice is acquired to obtain target voice. The target's perspective is the same as the human eye's first-person perspective. The target visual data and the target speech are sent to the data processing system, and the target data processing result sent by the data processing system is received, wherein the data processing system is the data processing system of claim 11 above.

13. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 10.

14. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method of any one of claims 1 to 10.

15. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method of any one of claims 1 to 10.

Images