Data processing method, apparatus, device, product, and storage medium

Abstract

Embodiments of the present application disclose a data processing method, device, equipment, product and storage medium. Embodiments of the present application obtain at least one original data, the original data comprising visual content and original description text corresponding to the visual content; identify a description text segment corresponding to at least one object in the visual content in the original description text; detect the semantic complexity of the description text segment to obtain a semantic detection result, the semantic detection result indicating the semantic complexity of the description text segment; perform mask processing on the description text segment based on the semantic detection result, and perform text restoration on the processed description text segment based on the visual content; determine the semantic similarity between the restored description text segment and the description text segment, and update the original data according to the semantic similarity. The scheme can accurately identify noise in the data and complete targeted correction, effectively improving the quality of the data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data processing method, apparatus, equipment, product, and storage medium. Background Technology

[0002] As artificial intelligence fully enters the era of multimodal large-scale models, the core understanding capabilities of models—whether for text-to-image, text-to-video, image-to-text, or visual understanding—all rely heavily on massive and high-quality "visual content-text description" data pairs. These data pairs are the cornerstone of model learning cross-modal semantic alignment. Only with precisely mapped image and text data can models deeply understand the relationship between "low-level pixel features" and "high-level natural language semantics," thereby possessing accurate visual perception capabilities and realistic content generation abilities. In the actual large-scale training data collection and annotation process, due to the accuracy error of automated algorithms or the negligence of manual annotation, a large amount of noisy data with inconsistent text and images is inevitably mixed into the training dataset. If these defective data are used directly for training, it will have an irreversible negative impact on the large model: (1) If the text describes an object that does not exist in the picture, the model will learn the wrong mapping relationship, resulting in the generation of non-existent content during inference; (2) If the text mislabels "red car" as "blue car", it will cause the model to confuse fine-grained attributes such as color and texture, resulting in the loss of control over the generation details; (3) For video data, if the action sequence described in the text does not match the actual time sequence of the picture, it will cause the model to be unable to learn the correct causal relationship and time logic.

[0003] Currently, to identify noise in the training data of large models, a comprehensive scoring method such as CLIP (Contrastive Language-Image Pre-training) / BLIP (Bootstrapping Language-Image Pre-training) is often used. This method encodes images / videos and entire text segments into vectors, calculates the cosine similarity between the two vectors, and considers them inconsistent if the score is below a certain threshold. Alternatively, a verification method based on VQA (Visual Question Answering) is used, where the text is transformed into a series of questions, which are then input into a VQA model to obtain the answers and compared with the text to identify erroneous text. Another method relies on human review of the video and reading of the text to manually mark erroneous text.

[0004] However, during the research and practice of existing technologies, the inventors of this application discovered that the overall scoring method is prone to confusing attributes when dealing with complex scenarios, resulting in situations where the image and text are inconsistent but the overall similarity score is still high, making it impossible to identify erroneous text; the output of the VQA model is usually open or a short classification label, which leads to a semantic gap between the VQA response and the original text, making it difficult to directly compare and identify erroneous text; when watching videos and reading text manually, there is not only a strong subjectivity leading to inconsistent recognition standards, but also a lack of expertise in accurately judging certain implicit or statistical features. Therefore, it is impossible to accurately identify noise in the data, and thus impossible to correct the noise, resulting in a decline in data quality. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, product, and storage medium that can improve data quality.

[0006] This application provides a data processing method, including: Obtain at least one piece of raw data, which includes visual content and the original descriptive text corresponding to the visual content; Identify the descriptive text fragment corresponding to at least one object in the visual content from the original descriptive text; Semantic recognition is performed on the descriptive text fragments to obtain the semantic logical structure information of the descriptive text fragments; Based on semantic logical structure information, at least one associated text word is identified in the descriptive text fragment. The associated text word is a text word used to describe the attribute information of the object. Based on the number of text words, the number of related text words, and the semantic logical structure information of the descriptive text fragment, the semantic complexity of the descriptive text fragment is detected, and the semantic detection result is obtained. The semantic detection result indicates the semantic complexity of the descriptive text fragment. Based on the semantic detection results, the descriptive text fragments are masked, and based on the visual content, the processed descriptive text fragments are restored. Determine the semantic similarity between the restored descriptive text fragments and the original descriptive text fragments, and update the original data based on the semantic similarity.

[0007] Accordingly, embodiments of this application also provide a data processing apparatus, including: An acquisition unit is used to acquire at least one piece of raw data, which includes visual content and the original descriptive text corresponding to the visual content. The recognition unit is used to identify the descriptive text fragment corresponding to at least one object in the visual content in the original descriptive text; The detection unit is used to perform semantic recognition on the descriptive text fragment to obtain the semantic logical structure information of the descriptive text fragment. It is also used to identify at least one associated text word in the descriptive text fragment based on the semantic logical structure information. The associated text word is a text word used to describe the attribute information of the object. It is also used to detect the semantic complexity of the descriptive text fragment based on the number of text words in the descriptive text fragment, the number of associated text words, and the semantic logical structure information, and obtain the semantic detection result. The semantic detection result indicates the semantic complexity of the descriptive text fragment. The restoration unit is used to mask the descriptive text fragments based on the semantic detection results and restore the processed descriptive text fragments based on the visual content. The update unit is used to determine the semantic similarity between the restored descriptive text fragment and the original descriptive text fragment, and to update the original data based on the semantic similarity.

[0008] In some embodiments, the restoration unit can be specifically used to determine the semantic complexity type of the descriptive text fragment based on the semantic detection result. The semantic complexity type includes either a simple description or a complex description. When the semantic complexity type is a complex description, the unit identifies the text word position of the associated text word in the descriptive text fragment and determines the attribute type corresponding to the associated text word. Based on the text word position, the unit adds a text mask corresponding to the attribute type to the descriptive text fragment to obtain at least one processed descriptive text fragment.

[0009] In some embodiments, the restoration unit may further be used to: filter out the current visual content corresponding to the descriptive text fragment from the visual content according to the content type of the visual content; perform text prediction on the text mask in the processed descriptive text fragment based on the current visual content to obtain the predicted text word corresponding to the text mask; and replace the text mask with the predicted text word in the processed descriptive text fragment to obtain the restored descriptive text fragment.

[0010] In some embodiments, the restoration unit may further be used to perform text prediction on the text mask in the processed descriptive text fragment based on the current visual content using a text restoration model; when the text restoration model cannot recognize the current visual content, or when the accuracy of the text prediction result does not reach a preset accuracy threshold, attribute reference information of the attribute type corresponding to the text mask is obtained; based on the attribute reference information and the current visual content, the text restoration model is used to perform text prediction on the text mask in the processed descriptive text fragment to obtain the predicted text word corresponding to the text mask.

[0011] In some embodiments, the restoration unit can also be used to, when the content type is video content, filter out at least one video frame corresponding to the descriptive text fragment in the visual content to obtain the current visual content; and when the content type is image content, filter out the pixel region corresponding to the descriptive text fragment in the visual content to obtain the current visual content.

[0012] In some embodiments, the data processing apparatus further includes a comparison unit, which can be specifically used to: filter out the current visual content corresponding to the descriptive text fragment in the visual content when the semantically complex type is a simple description; use a semantic comparison model to perform a semantic comparison between the descriptive text fragment and the current visual content to obtain a semantic comparison result; filter out at least one descriptive text fragment whose semantic comparison result is different from the descriptive text fragment to obtain the descriptive text fragment to be updated; and update the descriptive text fragment to be updated according to the current visual content to update the original data.

[0013] In some embodiments, the updating unit can be specifically used to filter out at least one descriptive text fragment whose semantic similarity does not exceed a preset similarity threshold from the descriptive text fragments to obtain the descriptive text fragment to be updated; in the original data, the descriptive text fragment to be updated is replaced with the corresponding restored descriptive text fragment to obtain the updated data.

[0014] In some embodiments, the data processing apparatus further includes a prompting unit, which can be specifically used to compare the descriptive text fragment to be updated with the corresponding restored descriptive text fragment; based on the comparison result, identify at least one text word to be updated in the descriptive text fragment to be updated, and identify the restored text word corresponding to the text word to be updated in the restored descriptive text fragment; and generate update prompt information corresponding to the updated data based on the text word to be updated and the restored text word.

[0015] In some embodiments, the identification unit may be used to perform semantic recognition on the original descriptive text to obtain the semantic logical structure information of the original descriptive text; based on the semantic logical structure information of the original descriptive text, identify at least one object in the visual content of the original descriptive text; and extract the text content corresponding to the object from the original descriptive text to obtain a descriptive text fragment.

[0016] Furthermore, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to perform the steps in the data processing method provided in embodiments of this application.

[0017] Furthermore, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in the data processing method provided in embodiments of this application.

[0018] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the data processing methods provided in embodiments of this application.

[0019] In this embodiment, after acquiring at least one piece of raw data, a descriptive text fragment corresponding to at least one object in the visual content is identified in the raw descriptive text. Then, semantic recognition is performed on the descriptive text fragment to obtain its semantic logical structure information. Based on the semantic logical structure information, at least one associated text word is identified in the descriptive text fragment. The associated text word is a text word used to describe the attribute information of the object. Based on the number of text words in the descriptive text fragment, the number of associated text words, and the semantic logical structure information, the semantic complexity of the descriptive text fragment is detected to obtain a semantic detection result. Then, based on the semantic detection result, the descriptive text fragment is masked, and based on the visual content, the processed descriptive text fragment is restored. Finally, the semantic similarity between the restored descriptive text fragment and the original descriptive text fragment is determined, and the raw data is updated based on the semantic similarity.

[0020] This scheme decomposes the original descriptive text into descriptive text fragments corresponding to objects in the visual content, transforming the entire text into fine-grained semantic units. This avoids the problems of existing overall scoring methods, such as easily confused attributes due to coarse-grained comparison and inability to locate specific erroneous text. Furthermore, it performs semantic complexity detection on the descriptive text fragments, masks them based on the semantic detection results, and restores the processed descriptive text fragments using the visual content as objective factual basis. By reconstructing the text content based on real visual features, it eliminates the semantic comparison gap and the subjectivity and instability inherent in manual review. Additionally, by determining the semantic similarity between the restored descriptive text fragments and the original descriptive text fragments, it can accurately locate noisy text that does not match the visual content. Then, it updates and corrects the original data based on the similarity results, thereby accurately identifying noise in the data and performing targeted corrections, effectively improving data quality. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1a This is a schematic diagram of a scenario illustrating the data processing method provided in an embodiment of this application; Figure 1bThis is a flowchart illustrating the data processing method provided in an embodiment of this application; Figure 2a This is another schematic flowchart of the data processing method provided in the embodiments of this application; Figure 2b This is a schematic diagram of the terminal architecture provided in the embodiments of this application; Figure 2c This is a schematic diagram of the disassembly module provided in the embodiments of this application; Figure 2d This is a schematic diagram of the semantic complexity detection module provided in the embodiments of this application; Figure 2e This is a schematic diagram of the operation of the mask module provided in the embodiments of this application; Figure 2f This is a schematic diagram of the semantic similarity discrimination module provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] This application provides a data processing method, apparatus, device, product, and storage medium. Specifically, the data processing apparatus can be integrated into an electronic device, such as a terminal or server.

[0025] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0026] For example, see Figure 1a , Figure 1a This is a schematic diagram illustrating a scenario of the data processing method provided in an embodiment of this application. For example... Figure 1a As shown, taking an example where a data processing device is integrated into an electronic device, and the electronic device is a terminal, the terminal acquires at least one piece of raw data, including visual content and corresponding raw descriptive text. Then, it identifies descriptive text fragments corresponding to at least one object in the visual content within the raw descriptive text. Next, it performs semantic detection on the descriptive text fragments to obtain semantic detection results, which indicate the semantic complexity of the descriptive text fragments. Then, based on the semantic detection results, it performs masking processing on the descriptive text fragments and performs text restoration based on the visual content. Finally, it determines the semantic similarity between the restored descriptive text fragments and the original data based on the semantic similarity, thereby improving data quality.

[0027] It is understood that, in the specific embodiments of this application, data and other related data are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0028] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0029] This embodiment will be described from the perspective of a data processing device, which can be integrated into an electronic device, such as a server or a terminal.

[0030] A data processing method, comprising: The process involves: acquiring at least one set of raw data, including visual content and corresponding raw descriptive text; identifying descriptive text fragments corresponding to at least one object in the visual content from the raw descriptive text; performing semantic recognition on the descriptive text fragments to obtain their semantic logical structure information; identifying at least one associated text word in the descriptive text fragments based on the semantic logical structure information, where the associated text word is a text word used to describe the attribute information of the object; detecting the semantic complexity of the descriptive text fragments based on the number of text words, the number of associated text words, and the semantic logical structure information, and obtaining a semantic detection result indicating the semantic complexity of the descriptive text fragments; masking the descriptive text fragments based on the semantic detection result, and restoring the processed descriptive text fragments based on the visual content; determining the semantic similarity between the restored descriptive text fragments and the original data based on the semantic similarity.

[0031] For example, see Figure 1b , Figure 1b This is a flowchart illustrating the data processing method provided in an embodiment of this application. Figure 1b As shown, the specific process of this data processing method is as follows: 101. Obtain at least one piece of raw data.

[0032] The raw data includes visual content and the original descriptive text corresponding to the visual content.

[0033] The original data can be the initial dataset used by a multimodal large-scale model, such as a text-to-image model, a text-to-video model, an image-to-text model, or a visual understanding model. It should be noted that when training a multimodal large-scale model, the original data can be the original training data.

[0034] The original training data is the initial dataset used during model training. It serves as the foundation for the model's learning and training, containing various information related to the target task. This information is diverse; depending on the model task and application scenario, the original training data can include numerical data, text data, image data, audio data, etc. By learning from the original training data, the model builds its understanding and execution capabilities for the task, such as prediction, classification, and clustering. In machine learning and deep learning, the quality and quantity of the original training data have a crucial impact on model performance. High-quality, large-scale original training data helps the model learn more accurate and comprehensive patterns and rules, thereby improving the model's performance in practical applications. It's worth noting that the original training data has cross-modal characteristics; it associates and combines data from different modalities, allowing the model to learn the inherent connections and semantic alignment between different modalities. Common modalities of multimodal data include visual, text, and audio. The original training data integrates these different modalities to provide rich information input to the model, enabling the model to understand and process information from multiple perspectives and improve its ability in complex tasks, such as image-to-text, text-to-image, or multimodal question answering. The accurate correlation and richness of different modalities in the original training data directly affect the model's generation quality and answer accuracy.

[0035] The original training data can be of various types. For example, it can include data pairs consisting of visual content and corresponding descriptive text. These data pairs are crucial for the model to learn cross-modal semantic alignment. For instance, when training a text-to-image generation model, the original training data can include a large number of images and detailed text describing these images. The images serve as the visual content, while the text describes the objects, scenes, actions, etc., within the images. The visual content and text descriptions in these data pairs need to correspond accurately for the model to learn the correct image-text mapping relationship. If there are inconsistencies between the images and text—for example, if an image shows a cat but the text describes a dog—the model will learn incorrect information, affecting its performance in the generation task.

[0036] Visual content can be understood as the information carrier that can be received and processed by the visual perception system. In the field of computer vision, visual content is the foundational data for tasks such as object recognition, image classification, and video understanding. Within multimodal data, visual content, as an important modality, works in conjunction with other modalities (such as text, audio, etc.) to provide the model with multidimensional information. It provides intuitive image or video information for the model to understand real-world scenes, objects, and events. By extracting and analyzing features from visual content, the model can understand the semantic information it contains and correlate and fuse it with data from other modalities, thereby achieving more powerful multimodal task processing capabilities. For example, in text-to-image and image-to-text models, visual content refers to the image or video frame sequence portion of the original training data. When processing images, each pixel in the image carries information such as color and brightness. This information is combined to form the image content we see. For example, in an image depicting a landscape, the shape, color, and position of objects such as mountains, rivers, and trees are all part of the visual content. For a video frame sequence, each video frame is an image. The video displays dynamic scenes by playing these frames continuously. For example, in a video recording a sports meet, the changing information of the athletes' movements, the layout of the venue, and the audience's reactions over time in the video frame sequence are all part of the visual content.

[0037] The original descriptive text corresponding to the visual content is the initial textual information that semantically represents the visual content. The original descriptive text uses the vocabulary, grammar, and semantic structure of natural language to describe and explain objects, scenes, actions, and relationships within the visual content. Accurate original descriptive text guides the model to correctly understand the information conveyed by the visual content; conversely, incorrect or inaccurate descriptive text leads the model to learn incorrect cross-modal mapping relationships. Ensuring consistency between the original descriptive text and the visual content is crucial in model training and application. For example, taking image-text data pairs in large model training as an example, the original descriptive text corresponding to the visual content is a textual description written for an image or video frame sequence, which can take the form of a title, detailed description, subtitle, or script. For instance, for an image showing children playing in a garden, the corresponding original descriptive text could be, "A group of lovely children are playing happily in a flower-filled garden; some are chasing butterflies, and some are playing with flowers." This text details the subject (children), the scene (garden), and the subjects' actions (chasing butterflies, playing with flowers) in the image. During the training of a large model, the model attempts to learn the semantic relationship between the original descriptive text and the corresponding image so that it can accurately generate the target content in subsequent tasks, such as generating images from text or generating text from images. Therefore, it is necessary to perform consistency checks on the visual content and its corresponding original descriptive text, and update the descriptive text that is inconsistent with the visual content in order to improve the quality of the training data.

[0038] When acquiring raw data, data can be read from a database storing multimodal data. This database contains a large number of data pairs consisting of visual content and corresponding original descriptive text. By writing database query statements, one or more such data pairs can be retrieved from the database as raw data according to certain conditions (such as data category, time range, etc.).

[0039] 102. Identify the descriptive text fragment corresponding to at least one object in the visual content within the original descriptive text.

[0040] In this context, an object can be understood as a specific thing, concept, or entity involved in a particular scene or text description. It's important to note that in multimodal data processing, especially in scenarios involving the association of text and visual content, objects are the key element connecting the semantics of the two. The text description aims to elucidate object-related information within the visual content, while the visual content presents these objects in the form of intuitive images or videos. The model needs to identify objects in the visual content and accurately match them with objects in the text description, thereby achieving cross-modal semantic alignment. Objects not only include concrete, visible objects but may also involve abstract concepts, events, etc. For example, an object can be a describable subject in the visual content, and its type can include concrete objects, such as "trees," "bench," and "pedestrians" in a picture of a park scene; it can also be an abstract concept, such as the event of "football match" in a video about two teams playing a football match, which can be considered an object.

[0041] Descriptive text fragments are relatively independent semantic parts obtained by breaking down complete text according to certain logic or rules. Complete text typically contains rich information; to analyze and process it more meticulously, it needs to be broken down into multiple descriptive text fragments. These fragments can be phrases, sentences, or specific text blocks, each expressing partial semantic information about an object, event, or scene. It's important to note that descriptive text fragments can be text units that provide detailed descriptions of relevant information about objects in visual content. They are closely related to the objects in the visual content and are used to describe the object's attributes, actions, states, relationships, and other characteristics. Descriptive text fragments can take many forms. For example, they can include text presented as "entity-attribute-relationship" triples, such as "<cat, color, black>" or "<person, action, running>". They can also be simple phrases describing a feature of an object, such as in the text describing a landscape image, "Under the azure sky, there is a patch of emerald green grass," where "azure sky" and "emerald green grass" are descriptive text fragments, describing the color characteristics of the sky and grass respectively. Alternatively, they can be multiple phrases describing a specific object, such as in the text describing a person, "The girl stands on a brown wooden floor with natural texture. She has fair skin and pink legs. She is wearing pure white knee-high socks and beige low-top casual shoes." The descriptive text fragments corresponding to the object "girl" include "standing on a brown wooden floor with natural texture," "fair skin," "wearing pure white knee-high socks," and "wearing beige low-top casual shoes."

[0042] When identifying descriptive text fragments corresponding to at least one object in the visual content within the original descriptive text, the text structure can be analyzed based on syntactic analysis and semantic understanding models to determine the descriptive part corresponding to each object. Alternatively, the original descriptive text can be input into a large model, and then prompts with splitting rules can be input into the large model to split the original descriptive text into objects and their corresponding descriptive text fragments based on the semantic logical structure information in the original descriptive text, such as subject, verb, and object.

[0043] One way to split the original description text into objects and their corresponding description text fragments is as follows: For example, semantic recognition is performed on the original descriptive text to obtain the semantic logical structure information of the original descriptive text; based on the semantic logical structure information of the original descriptive text, at least one object in the visual content is identified in the original descriptive text; the text content corresponding to the object is extracted from the original descriptive text to obtain the descriptive text fragment.

[0044] Semantic recognition is the process of analyzing text content to understand its meaning, concepts, and semantic relationships between its parts. Semantic logical structure information can be understood as the organizational structure and logical relationships presented by the text at the semantic level, reflecting the interrelationships and dependencies between various semantic elements. Semantic logical structure information not only includes grammatical relationships between words but also emphasizes logical connections at the semantic level, such as causal relationships, purpose relationships, parallel relationships, and modification relationships. By analyzing and understanding semantic logical structure information, we can gain a deeper understanding of the text's meaning, clarify its content, and identify the logical connections between its parts.

[0045] For example, when performing semantic recognition on the original descriptive text, the original descriptive text can be input into a large language model, and preset splitting rule prompts can be provided. The large language model analyzes the vocabulary, grammar, and semantic relationships in the text through the splitting rule prompts, and sorts out the internal logical structure of the text. For example, it determines which words constitute the subject, predicate, and object, which words are modifiers and who they modify, and so on, thereby outputting semantic logical structure information that can reflect the logical relationships in the text.

[0046] In this context, splitting rule prompts can be understood as guiding information used to instruct the large language model to split the original descriptive text according to specific rules. Splitting rule prompts are key instructions for breaking down complex original descriptive text into multiple independent semantic parts (i.e., descriptive text fragments), helping to improve the accuracy and standardization of data processing. It's important to note that original descriptive text is usually a complete and complex natural language expression. To accurately split it into descriptive text fragments corresponding to each object, clear and reasonable rules are needed. By inputting splitting rule prompts into the model, the model can accurately split the text according to established rules, avoiding interference between information from different independent semantic parts. Splitting rule prompts can take various forms; for example, they can include splitting rule prompts based on grammatical structures. For example, taking a sentence containing a subject-verb-object structure as the original descriptive text, the splitting rule prompt could be "According to the subject-verb-object structure, the subject and related modifiers are treated as one subject, and the predicate and related descriptions of the object are treated as descriptive text fragments of that subject." Then, for the sentence "Diligent workers carefully build sturdy houses," the model can identify "diligent workers" as the subject and "carefully build sturdy houses" as the corresponding descriptive text fragment based on this prompt. The splitting rule prompt can also include semantically logical splitting rule prompts, such as "Taking an object with complete semantics as the core, all descriptive content surrounding that object is treated as its descriptive text fragment." Then, for the text "Colorful flowers bloom in the park, and there is a clear lake nearby," the model can, based on this prompt, treat "flowers" as one object, "blooming in colorful flowers" as the descriptive text fragment corresponding to the flowers, and "lake" as another object, with "there is a clear lake nearby" as the descriptive text fragment corresponding to the lake.

[0047] Specifically, based on the subject-verb-object relationships presented in the semantic logical structure information, the object in the sentence is identified as the agent of the action or the thing being described. After identifying the object, all descriptive content related to that object is extracted from the original descriptive text based on the semantic logical structure information, thus forming a descriptive text fragment. The descriptive content may include the object's attributes, actions, relationships with other things, etc.

[0048] This solution obtains semantic logical structure information from the original descriptive text through semantic recognition, enabling a deeper understanding of the text's internal logic. Based on this semantic logical structure information, it accurately identifies objects within the visual content and further extracts the corresponding descriptive text fragments, refining the complex original descriptive text into more easily processed units. Since the attention span of multimodal models is limited, the longer the text description, the less attention each individual word receives during processing, resulting in lower accuracy. Therefore, to improve the coverage and accuracy of text description processing, at least one subject is identified from the original descriptive text, and the original descriptive text is broken down into descriptive text fragments corresponding to that subject. This allows the multimodal model to focus its attention on a limited number of words when processing these fragments, thus improving the model's coverage and accuracy.

[0049] 103. Perform semantic detection on the descriptive text fragments and obtain the semantic detection results.

[0050] Semantic detection can be understood as the process of analyzing, understanding, and evaluating the meaning expressed by a text. It involves determining the semantic relationships between various elements in the text, such as the relationship between subject and object, action and object, and understanding the intention and emotion conveyed by the text. It should be noted that when performing semantic detection on a descriptive text fragment, the detection can focus on the structure of the text within the fragment. The semantic detection results can determine the complexity of the descriptive text fragment.

[0051] The semantic detection results can take various forms. For example, they can include information indicating the semantic complexity of a text fragment. Taking the text fragment describing "a cat with smooth fur lying quietly on a soft cushion" as an example, the semantic detection results show that the text fragment has a high degree of semantic complexity because it contains multiple modifiers ("smooth fur", "soft") and relatively complex action descriptions ("lying quietly").

[0052] Semantic complexity can be understood as an indicator of the semantic richness, structural complexity, and comprehension difficulty of a text. The semantic complexity of a text is influenced by various factors, including the richness of its vocabulary, the complexity of its sentence structure, the depth of nested logical relationships, and semantic ambiguity. Simple texts typically have clear semantics, fewer modifiers, and direct logical relationships, making them easy to understand; while complex texts may contain a large number of specialized terms, long and difficult sentences, multiple modifiers, and complex logical reasoning, requiring more language knowledge and processing skills to accurately understand their meaning. For example, the semantic complexity of the text describing "the sun rises" is relatively low because its vocabulary is simple, its structure is direct, and it only expresses a clear action; while the semantic complexity of "the huge sun, radiating golden light, slowly rose from the sea, illuminating the entire calm and shimmering sea" is higher, as it contains multiple modifiers and has a relatively complex sentence structure.

[0053] Semantic detection of descriptive text fragments involves processing multiple levels of the text, including vocabulary, grammar, semantic structure, and contextual relationships. This allows for a comprehensive and accurate understanding of the semantic complexity of the text fragment. The semantic detection results can be obtained by performing semantic detection on descriptive text fragments as follows: For example, semantic recognition is performed on a descriptive text fragment to obtain its semantic logical structure information; based on the semantic logical structure information, at least one associated text word is identified in the descriptive text fragment, which is a text word used to describe the attribute information of an object; based on the number of text words in the descriptive text fragment, the number of associated text words, and the semantic logical structure information, the semantic complexity of the descriptive text fragment is detected to obtain the semantic detection result.

[0054] There are several ways to perform semantic recognition on descriptive text fragments. For example, syntactic analysis trees can be used to display the grammatical and semantic relationships between words, thereby uncovering their logical structure. Semantic role labeling technology can also be used to determine the role each word plays at the semantic level, thus clarifying the semantic logical relationships. For instance, for the descriptive text fragment "The cute little cat was chasing a colorful ball with some pretty patterns on it," the semantic logical structure information is as follows: "the little cat" is the subject, and "cute" modifies the cat; "chasing a colorful ball" describes the cat's action and the object of the action, where "colorful" modifies "ball"; "the ball has some pretty patterns on it" further explains the characteristics of the ball, and is parallel to the previous description of the ball.

[0055] In this context, related text words can be understood as words in a text that are semantically closely connected to a specific object or topic and are used to further describe or define the relevant characteristics of that object or topic. When analyzing text fragments describing visual content, related text words are used to clarify the specific characteristics of the visual object, which correspond to the actual situation in the visual content.

[0056] In this context, an object's attribute information can be understood as information describing the object's characteristics, properties, state, quantity, and so on. There are various types of object attribute information, including not only common physical attributes such as shape, color, size, and quantity, but also action attributes, material attributes, and so on.

[0057] Specifically, based on semantic logical structure information, when at least one related text word is identified in a descriptive text fragment, the analysis of the semantic organization and logical relationships of the text fragment identifies words in the text fragment that are semantically closely connected to a specific object and used to describe the object's related features. Semantic logical structure information reveals the way the text describes visual objects and the logical relationships between them. Through this structure, it is possible to determine which words are related text words used to describe the object's attributes, state, and other features.

[0058] There are several ways to identify related text words. For example, semantic role labeling can be used to label the role that each word in the text plays at the semantic level, thereby determining which words are associated with a specific object and describe its attributes. Alternatively, knowledge graphs can be used to help identify related text words in the text by utilizing the relational information about objects and their attributes in the knowledge graph.

[0059] The number of words describing a text fragment is the total number of words contained in a text fragment with relatively independent semantics. It reflects, to some extent, the richness of information conveyed by the text; the more words, the richer the information. It should be noted that when performing semantic complexity analysis on text fragments describing visual content, a larger number of words may mean that the text fragment contains richer details and a more comprehensive description of the visual object, which may lead to increased semantic complexity.

[0060] The number of related text words reflects the level of detail and richness of the description of a specific object. Related text words revolve around the object, depicting it from different angles, making the object's image and characteristics more concrete and clear in the text. The number of related text words influences the semantic complexity of the text to some extent; more related text words mean a more detailed description of the object, potentially making the text more semantically complex. For example, in the text describing a landscape image, "Several snow-white clouds of various shapes float in the azure sky, and sunlight shines through the thin clouds onto the lush green earth," words like "azure," "snow-white," "various shapes," "thin," and "lush green" are all related text words. They describe the sky, clouds, and earth from multiple aspects such as color, shape, and texture.

[0061] Semantic complexity is an indicator that measures the semantic complexity of a text fragment with relatively independent semantics. It comprehensively reflects the richness of the meaning expressed by the text, the ease of understanding, and the complexity of the semantic structure. Semantic complexity is influenced by a variety of factors, including the specialization of vocabulary, the complexity of sentence structure, the depth of nesting of logical relationships, and the richness of modifiers.

[0062] The semantic complexity of a descriptive text fragment is assessed by considering the number of words in the fragment, the number of related words, and the semantic logical structure. This involves comprehensively evaluating the fragment's semantic complexity based on the number of words, the number of descriptive words related to the object, and the semantic organization and logical relationships. The assessment results can be presented numerically or as a score. For example, when assessing the semantic complexity of a descriptive text fragment, quantitative indicators and rules are set, weights are assigned to the number of words and related words, and a score is awarded based on the complexity of the semantic logical structure, ultimately yielding the semantic detection result.

[0063] This solution employs semantic recognition to obtain semantic logical structure information from descriptive text fragments, enabling precise understanding of the text's semantic organization and logical connections, thus avoiding the one-sidedness of judging complexity solely based on surface features. By identifying associated text words describing object attributes based on this semantic logical structure information, it can accurately locate attribute description elements within the text, clarifying the basis for complexity judgment. Furthermore, by comprehensively detecting semantic complexity by combining the number of text words in the descriptive text fragment, the number of associated text words, and semantic logical structure information, it can quantify and objectively determine the degree of text complexity, forming standardized complexity grading results. This avoids the problems of redundant processing of simple texts and detection bias caused by inaccurate judgments of complex texts, while also improving the accuracy and objectivity of semantic complexity detection.

[0064] 104. Based on the semantic detection results, mask the descriptive text fragments and restore the text based on the visual content.

[0065] Masking, in this context, can be understood as the operation of covering or hiding certain parts of data in a specific way. Masking is not simply deleting parts of text; rather, it hides key information in a controlled manner while preserving context. Masking retains the overall structure and most of the semantic information of the text while hiding key attribute information. There are various types of masks; for example, key information can be randomly selected for masking, or masking can be performed according to preset rules (such as prioritizing the masking of descriptive words). Masking descriptive text fragments based on semantic detection results involves selectively masking parts of the text fragment based on the results of semantic analysis. Semantic detection results provide information about the semantic complexity and key information of the text fragment, making masking operations more purposeful and reasonable.

[0066] The processed descriptive text fragment is a new text fragment obtained by processing the original descriptive text fragment with specific techniques (such as masking). The processed descriptive text fragment retains the basic structure and some semantic information of the original text, while highlighting key information through masking. For example, the original descriptive text fragment is "A little boy is playing on a green grass with a red balloon in his hand." After masking, the key information related to color is masked, resulting in the processed descriptive text fragment "A little boy is playing on a grass with a [color MASK] balloon in his hand" (where "MASK" represents a mask). This processed descriptive text fragment retains the basic description of the little boy's actions and the scene, while highlighting the two key attributes of color through masking.

[0067] Text restoration is the process of recovering or regenerating the altered text after certain changes have been made (such as masking or deleting parts of the text). It can restore masked descriptive text fragments based on visual content. Since masking hides key information in descriptive text fragments, information from the visual content can be used to fill in these missing key information, thereby obtaining a complete text closely related to the visual content.

[0068] When restoring text from processed descriptive text fragments based on visual content, a large model can be used to identify the attribute information of objects in the image, extract relevant visual features based on image recognition technology, and then generate corresponding text by combining natural language processing capabilities to restore the masked part.

[0069] The method for masking descriptive text fragments based on semantic detection results can be as follows: For example, based on the semantic detection results, the semantic complexity type of the descriptive text fragment is determined. The semantic complexity type includes either simple description or complex description. When the semantic complexity type is complex description, the text word position of the associated text word is identified in the descriptive text fragment, and the attribute type corresponding to the associated text word is determined. Based on the text word position, a text mask corresponding to the attribute type is added to the descriptive text fragment to obtain at least one processed descriptive text fragment.

[0070] Semantic complexity types are categories derived from classifying the semantic complexity of text. Different semantic complexity types reflect differences in vocabulary, grammar, semantic structure, and comprehension difficulty. For example, simple text may have straightforward semantics, while complex text may contain multiple layers of logical relationships, rich modifiers, or specialized vocabulary. For text types with low semantic complexity, relatively simple and direct verification methods can be used; however, for text types with high semantic complexity, more refined and complex verification processes are required. The semantic complexity type can be determined by comprehensively analyzing factors such as the number of words in the descriptive text fragment, the number of related words, and the semantic logical structure. For example, descriptive text fragments with few words, few related words, and simple semantic logic can be classified as simple descriptive types; while descriptive text fragments containing rich modifiers, a large number of words, and complex semantic logic should be classified as complex descriptive types.

[0071] In this context, a simple description can be understood as a concise and direct semantic explanation of an object, event, or scene in textual expression. Such descriptions typically use basic, common vocabulary, have simple sentence structures, clear logical relationships, and convey relatively singular, easily understood information.

[0072] Complex descriptions can be understood as employing rich vocabulary, complex sentence structures, and diverse logical relationships in textual expression to provide a comprehensive, detailed, and in-depth semantic exposition of objects, events, scenes, etc. Such descriptions often contain numerous modifiers, qualifiers, or nested logical relationships to convey richer and more precise information.

[0073] The semantic detection results include information about the vocabulary, grammar, semantic structure, and logical relationships of the text fragment. By comprehensively considering and analyzing this information, the semantic complexity of the text fragment can be determined, and the semantic complexity type can be identified based on the semantic complexity.

[0074] The text word position of a related text word refers to the specific location within a text sequence of words that have a close semantic connection to a particular object and are used to describe the characteristics of that object. Determining the text word position can be achieved using techniques such as part-of-speech tagging and dependency parsing. First, part-of-speech tagging identifies the related text words, and then dependency parsing is used to determine their positions within the sentence structure, thus accurately obtaining the text word position of the related text words.

[0075] Among them, the attribute types corresponding to related text words are the categories of object features represented by descriptive words closely related to a specific object. Different related text words are used to characterize different aspects of the object's features, which can be summarized into various attribute types. Attribute types help classify and understand the features of objects in text. Attribute types can include color, shape, size, material, action, state, quantity, etc. It should be noted that in text describing visual content, the attribute types corresponding to related text words should match the attribute types actually presented by the object in the visual content. The attribute types corresponding to related text words can be determined through techniques such as semantic role labeling. For example, semantic role labeling can clarify the semantic role played by a word, thereby determining its corresponding attribute type.

[0076] Text masking involves replacing the original text with specific symbols, markers, or placeholders, making the masked information invisible or hidden on the surface. For example, text masks can use specific marker formats such as "[color MASK]" or "[shape MASK]". For the text fragment "That red car was driving on a wide road," to mask the related text words describing the car's color and road features, we can replace "red" with "[color MASK]" and "wide" with "[width MASK]", resulting in "That [color MASK] car was driving on [width MASK] road". This process preserves the overall structure and most of the semantic information of the text while hiding key attribute information. Adding text masks can be done using string manipulation functions in programming languages, accurately inserting corresponding mask markers into the text fragment based on the position of the text words.

[0077] This solution precisely categorizes descriptive text fragments into two semantic complexity types—simple and complex—based on semantic detection results. This allows for the separate processing of complex and simple text, avoiding redundant masking operations on simple descriptions and improving overall processing efficiency. When a semantic complexity type is determined to be a complex description, the solution accurately identifies the positions of related text words and determines their corresponding attribute types within the descriptive text fragment. This locks down the attribute elements used to describe the object in the text, ensuring the targeting and accuracy of the masking operation. Based on the text word positions, a text mask of the corresponding attribute type is added to the descriptive text fragment to obtain the processed descriptive text fragment. This fully preserves the semantic framework and contextual logic of the original text, avoiding the problems of semantic breaks caused by directly deleting key information and reduced relevance between the subsequent text restoration and the original text.

[0078] Among them, the method for restoring text from processed descriptive text fragments based on visual content can be as follows: For example, based on the content type of the visual content, the current visual content corresponding to the descriptive text fragment is selected from the visual content; based on the current visual content, text prediction is performed on the text mask in the processed descriptive text fragment to obtain the predicted text words corresponding to the text mask; the text mask is replaced with the predicted text words in the processed descriptive text fragment to obtain the restored descriptive text fragment.

[0079] The content type of visual content is determined through visually perceptible information. Visual content types include, but are not limited to, images, videos, and animations. For example, images, as a static visual content type, have relatively fixed information, including the spatial relationships and features of each element. Videos, as a dynamic visual content type, include not only information from each frame but also changes over time, such as the trajectory of objects and the continuity of actions. For instance, for a text fragment describing "a cat sleeping on the grass," if the visual content is a static image, it is image-type visual content. When reconstructing the text, one needs to find information such as the cat, the grass, and the cat's sleeping state within the image. If the visual content is a video, it is video-type visual content. When reconstructing the text, one needs to consider not only whether the video shows a cat sleeping on the grass but also the changes in the cat's movements between frames, and whether it remains in a sleeping state throughout. The content type of visual content can be determined through file format, data structure characteristics, etc.

[0080] In this context, the current visual content refers to the visual information that is closely related to the text segment being analyzed and is selected for matching or verification with that text segment. In image recognition, the current visual content can be a region in the image that contains an object; in video analysis, the current visual content can be a video frame within a specific time period corresponding to the text description.

[0081] When filtering the current visual content corresponding to a descriptive text fragment based on the content type of the visual content, the information closely related to the object, scene, or action described in the specific text fragment can be extracted from the overall visual data based on the classification results of the visual information. Since different types of visual content (such as images and videos) have different information presentation methods and characteristics, the filtering methods also differ. For image-type visual content, the spatial location and relevant object area in the image are mainly determined based on the text description; for video-type visual content, in addition to considering spatial information, the time dimension must also be considered to determine the time period and keyframes corresponding to the text description. For example, if the descriptive text fragment is "a vase contains a bouquet of colorful flowers," and the visual content is an image of an indoor scene, firstly, based on the characteristics of the image type, image recognition technology is used to detect and classify the objects in the image, identify the location of the vase and flowers, and determine the area containing the vase and flowers as the current visual content; if the visual content is a video showing an indoor arrangement, object detection and action recognition technology are used to find the time period in the video where the vase and flowers appear, and the video frames showing the flowers within that time period are taken as the current visual content. When filtering current visual content, computer vision algorithms or models can be used, such as locating relevant objects in image or video frames using a deep learning-based object detection model, or detecting the motion of objects in a video using optical flow methods, thereby accurately filtering the current visual content corresponding to the descriptive text fragment.

[0082] Text prediction, in particular, is the process of inferring and generating unknown text content based on given contextual information, language models, and relevant data. It can predict the most likely subsequent text or missing text based on existing text fragments, statistical patterns of language, and semantic understanding. It's important to note that when processing text related to visual content, text prediction relies not only on the text's own context but also on visual information to fill in the text mask in the masked descriptive text fragment—that is, inferring the original text content of the masked portion based on visual content. There are various methods for text prediction. For example, a pre-trained language model combined with image features can be used. The image is first converted into a feature vector, and then input into the pre-trained language model along with the processed descriptive text fragment. The model then predicts the most likely text at the text mask location based on learned language patterns and image information. Alternatively, a dedicated cross-modal prediction model can be constructed, integrating image understanding and text generation modules to directly generate predicted text from visual content and masked text. The predicted text words are specific words used to fill in the text mask in the processed descriptive text fragment based on visual content. These predicted text words must not only meet the linguistic requirements of the text itself, but also accurately reflect the information in the visual content.

[0083] For example, given the processed descriptive text fragment "The cup contains liquid of [liquid color MASK]", the current visual content is an image of a cup containing liquid. First, image recognition technology is used to identify the color of the liquid in the image, extracting color features, such as identifying the liquid color as "yellow". Then, this visual information is combined with the processed descriptive text fragment and input into a deep learning-based text prediction model. Based on learned language patterns and visual information, the model predicts "yellow" as the predicted text word, resulting in "The cup contains yellow liquid". During this process, the accuracy of the predicted text word can be improved by adjusting model parameters and increasing training data. If the predicted text word is inconsistent with the original descriptive text's description of the liquid color, the accuracy of the original descriptive text needs to be checked, or the reasons for the difference between the visual content and the text need to be further analyzed to ensure the consistency and accuracy of multimodal data.

[0084] The process of replacing the text mask with predicted text words in the processed text fragment involves masking the text fragment, predicting the masked text words based on relevant information, and then filling the original text mask positions with these predicted text words to generate a new text fragment that is complete and semantically relatively intact. This replacement can be achieved using string replacement functions in programming languages; for example, in Python, the `replace()` method can be used.

[0085] This solution selects the current visual content corresponding to the descriptive text fragment based on the content type of the visual content. This allows for precise location of the corresponding pixel region in an image or the corresponding video frame in a video for the text semantic unit, avoiding interference from full-scale visual content comparison and providing accurate and targeted objective visual evidence for text reconstruction. Based on this current visual content, text prediction is performed on the text mask in the processed descriptive text fragment to obtain predicted text words. This can rely on real visual features to complete the factual reconstruction of the mask content, eliminating subjective assumptions and model illusions in text prediction and ensuring a high degree of matching between predicted text words and visual content. The text mask is then replaced with the predicted text words in the processed descriptive text fragment to obtain the reconstructed descriptive text fragment. This fully preserves the semantic framework and contextual logic of the original text, achieving precise reconstruction of fine-grained semantics. This provides reliable standard text for subsequent semantic similarity comparison between the reconstructed text and the original text, as well as data noise correction. This effectively solves the defects of existing multimodal text reconstruction, such as fuzzy visual evidence, distorted semantic reconstruction, and inaccurate comparison benchmarks, improving the accuracy of text reconstruction and consistency detection.

[0086] The method for predicting the predicted text words corresponding to the text mask based on the current visual content and describing the text fragment after processing can be as follows: For example, based on the current visual content, a text reconstruction model is used to predict the text mask in the processed descriptive text fragment; when the text reconstruction model cannot recognize the current visual content, or when the accuracy of the text prediction result does not reach the preset accuracy threshold, attribute reference information of the attribute type corresponding to the text mask is obtained; based on the attribute reference information and the current visual content, the text reconstruction model is used to predict the text mask in the processed descriptive text fragment, and the predicted text words corresponding to the text mask are obtained.

[0087] Text reconstruction models are specifically designed to restore and reconstruct text that has been processed (e.g., masking, truncation) to remove information, based on given contextual information, external data, or specific conditions. These models learn language patterns, semantic relationships, and grammatical rules. By analyzing known parts of the text and combining them with relevant data or conditions, they predict and generate the missing parts, restoring the text to a complete and semantically sound state. It's important to note that text reconstruction models not only need to understand the semantic structure of the text but also extract key information from the current visual content and transform it into appropriate textual expressions to fill in the text mask positions. Text reconstruction models can be of various types, such as those built on deep learning architectures or recurrent neural networks. These models capture the sequential features of the text and, combined with visual information, predict and reconstruct the text mask. Alternatively, specialized cross-modal neural networks can be trained to take images and text as input and directly output the reconstructed text.

[0088] In this context, when using a text reconstruction model to predict text based on the current visual content, the text mask in the processed descriptive text fragment represents key information in the original descriptive text that needs to be verified using visual content. The text reconstruction model, based on the current visual content, can reconstruct this key information based on visual information. For example, if the processed descriptive text fragment is "The building's [material mask] looks sturdy," and the current visual content is an image of a building, firstly, the image is input into an image feature extractor based on a convolutional neural network to obtain visual features related to the building's material. Simultaneously, the processed descriptive text fragment is input into the text reconstruction model. By fusing visual and text features, the model predicts the text word corresponding to "material mask," such as "stone."

[0089] The inability of the text reconstruction model to recognize the current visual content can be understood as its inability to effectively extract valuable features or information from given text-related visual information to complete the prediction task of the text mask. This can be due to various reasons, such as the complexity of the visual content itself, the limitations of the model, and noise interference in the data. When the text reconstruction model cannot recognize the current visual content, it means that the model has difficulty converting visual information into semantic information that matches the text, thus affecting the accurate prediction of the masked part in the processed descriptive text fragment, hindering the text reconstruction process based on visual content. For example, when processing low-resolution images or video frames with complex background interference as the current visual content, the text reconstruction model may not be able to accurately extract key information related to the text, such as the attributes and actions of the object. For instance, the current visual content is a car picture with a peculiar shooting angle and a blurry image, and the processed descriptive text fragment is "The car's [color mask] is unique." When the text reconstruction model tries to recognize this image, it cannot accurately determine the car's color due to the blurriness, which is the situation where the text reconstruction model cannot recognize the current visual content. At this point, the model may not be able to provide reasonable predicted text words for "color mask", or the prediction may deviate significantly from the actual situation.

[0090] The text prediction results did not reach the preset accuracy threshold. This can be understood as the prediction results obtained after using the text prediction model to infer and generate unknown parts of the text failing to meet the pre-set standards in terms of accuracy and reliability. Due to the complexity of multimodal data, text prediction models may be affected by factors such as the ambiguity of visual content and the asymmetry between text and visual information, leading to inaccurate prediction results.

[0091] Attribute reference information is a set of relevant information used to help determine specific attributes of an object in the text. This information typically revolves around various attributes of the object, including but not limited to the values ​​of object attributes, the relationships between attributes, and the manifestation of attributes in different scenarios. By referring to this information, the model can more accurately understand the description of object attributes in the text, or make more reasonable inferences when faced with uncertain attribute descriptions. It should be noted that attribute reference information can be obtained by calling a dedicated model or tool corresponding to the specific attribute information of the object in the text to identify the current visual content. For example, if the text reconstruction model cannot identify the positional information in the current visual content, a dedicated model for positional recognition can be called to identify the current visual content and obtain the attribute reference information corresponding to the position.

[0092] Specifically, when using a text reconstruction model to predict text masks in processed descriptive text fragments based on attribute reference information and current visual content, the model combines reference information related to the attributes corresponding to the text mask with corresponding visual information to infer the masked part of the text and generate corresponding words.

[0093] In this solution, when the text reconstruction model cannot recognize the current visual content or the accuracy of the text prediction result does not reach the preset accuracy threshold, it specifically obtains attribute reference information corresponding to the attribute type of the text mask. This can accurately supplement the text reconstruction model's shortcomings in recognizing fine-grained visual attributes such as color, orientation, and quantity, effectively avoiding prediction bias and illusion problems caused by the model's own recognition limitations. By combining the attribute reference information with the current visual content and re-using the text reconstruction model for prediction, the predicted text words can completely match the real visual evidence of the visual content, greatly improving the accuracy and reliability of text prediction.

[0094] The method for filtering out the current visual content corresponding to the descriptive text fragment from the visual content based on the content type can be as follows: For example, when the content type is video content, at least one video frame corresponding to the descriptive text fragment is selected from the visual content to obtain the current visual content; when the content type is image content, the pixel region corresponding to the descriptive text fragment is selected from the visual content to obtain the current visual content.

[0095] Video content is a dynamic carrier of visual information. It displays various scenes, actions, and events over time by continuously playing a series of image frames, accompanied by audio (which may be absent in some cases). When the visual information is in video form, at least one image frame that accurately reflects the content of the descriptive text fragment is selected from the numerous image frames in the video. By selecting suitable video frames as the current visual content, the focus can be placed on the information corresponding to the descriptive text fragment, avoiding interference from irrelevant information in the video and improving processing efficiency and accuracy. There are various methods for selecting video frames. In addition to object detection and motion analysis, keyframe extraction algorithms can be used on the video content, combined with keywords in the text description, to select the video frames most relevant to the text from the keyframes.

[0096] Image content is a collection of visual information presented in a two-dimensional form. It expresses information such as various objects, scenes, colors, and shapes through the arrangement and combination of pixels. Compared to video content, images are static, capturing a visual scene at a specific moment. When the visual information is in the form of an image, based on the information conveyed by the descriptive text fragment, a specific region that accurately represents the descriptive text fragment is delineated from the numerous pixels of the entire image. When selecting pixel regions, semantic segmentation technology can be used to classify each pixel in the image, directly obtaining the pixel region corresponding to the object described in the text.

[0097] Specifically, when the content type is video content, this solution filters out at least one video frame corresponding to the descriptive text fragment to obtain the current visual content. This can accurately locate the temporal scene fragment corresponding to the text semantics in the video, avoiding the problems of redundant comparison of the entire video frame sequence and interference with temporal information, and meeting the verification requirements of the temporal attributes of video data. When the content type is image content, the solution filters out the pixel region corresponding to the descriptive text fragment to obtain the current visual content. This can accurately lock the visual region of the text semantic unit in the image and eliminate visual interference from irrelevant regions of the image.

[0098] 105. Determine the semantic similarity between the restored descriptive text fragment and the original descriptive text fragment, and update the original data based on the semantic similarity.

[0099] Among them, the restored descriptive text fragment is the descriptive text fragment that has been processed by masking and other methods. It is the text fragment obtained after restoring the hidden or changed parts.

[0100] Semantic similarity can be understood as an indicator of the degree of similarity between two text fragments at the semantic level. Since visual content and text descriptions in multimodal data need to maintain consistency, comparing the semantic similarity of the two after processing and restoration can accurately determine whether the original descriptive text accurately expresses the visual content. High semantic similarity means that the original descriptive text and the text restored based on the visual content are highly consistent semantically, indicating that the original description accurately reflects the visual content; low semantic similarity suggests a possible inconsistency between the text and the image, requiring inspection and correction of the original data. Semantic similarity can be calculated in various ways. For example, it can include word vector-based methods, mapping words in the text to a vector space and measuring the semantic similarity of text fragments by calculating the distance between vectors (such as cosine similarity); or it can use deep learning model-based methods, such as using a pre-trained language model to encode the text and then determining semantic similarity by comparing the encoded vectors.

[0101] For example, the original descriptive text fragment is "The puppy runs happily on the green grass." After masking and visual content-based reconstruction, the reconstructed descriptive text fragment is "The puppy runs happily on the emerald green grass." A cosine similarity calculation method based on word vectors is used. First, the words in the two text fragments are converted into word vectors, such as "puppy," "grass," "green," "emerald green," "happy," "happy," and "running," each corresponding to its own vector representation. Then, the overall cosine similarity of these vector combinations is calculated. If the similarity value is close to 1, it indicates that the two text fragments have high semantic similarity, and the original descriptive text accurately reflects the visual content. If the similarity value is low, such as below a certain preset threshold (e.g., 0.6), it indicates that the two have significant semantic differences, and the original descriptive text may be inconsistent with the visual content, requiring further analysis and correction.

[0102] The original data can be updated based on semantic similarity in the following ways: For example, at least one descriptive text fragment whose semantic similarity does not exceed a preset similarity threshold is selected from the descriptive text fragments to obtain the descriptive text fragment to be updated; in the original data, the descriptive text fragment to be updated is replaced with the corresponding restored descriptive text fragment to obtain the updated data.

[0103] Among them, the descriptive text fragments to be updated are text fragments with relatively independent semantics that are identified as needing to be adjusted or modified to conform to specific standards or actual situations. When the semantic similarity of the descriptive text fragments and their corresponding visual content is evaluated, if the semantic similarity of some text fragments does not meet the expected standard, they will be marked as descriptive text fragments to be updated.

[0104] In this approach, when certain descriptive text fragments in the original data are determined to need updating (i.e., the descriptive text fragments to be updated), and more accurate restored descriptive text fragments have been obtained, these restored text fragments are used to replace the corresponding descriptive text fragments to be updated in the original data. This generates new data, improving data quality and enabling the data to more accurately reflect the actual situation, thereby enhancing the performance and accuracy of the model trained on this data. For example, in the original training data, there is a text fragment describing an image as "There is a yellow cat in the tree in the picture." However, after semantic comparison and related processing with the image, it is determined that this is a descriptive text fragment to be updated. Through image-based analysis and text prediction, the restored descriptive text fragment is obtained as "There is a gray cat on the roof in the picture." In the original training data, this restored descriptive text fragment, "There is a gray cat on the roof in the picture," replaces the original descriptive text fragment "There is a yellow cat in the tree in the picture," thus obtaining updated training data. The replacement can be done using programming. For example, in Python, a data processing library can be used to read the file storing the original training data, locate the line containing the descriptive text fragment to be updated, replace the descriptive text fragment to be updated in that line with the restored descriptive text fragment, and finally save the modified data again to complete the update of the training data.

[0105] Specifically, descriptive text fragments with semantic similarity not exceeding a preset similarity threshold are selected as descriptive text fragments to be updated. This allows for the objective and accurate identification of noisy text that does not conform to the semantics of the visual content using quantitative standards, avoiding the subjectivity of manual judgment and the ambiguity of the overall scoring method, and ensuring the accuracy of noise identification. In the original data, the descriptive text fragments to be updated are directly replaced with restored descriptive text fragments obtained based on visual content reconstruction. Text correction is completed based on the true visual features, completely eliminating problems such as attribute errors and hallucinatory descriptions in the original text. This achieves noise detection and automatic correction, and completes accurate data optimization without manual intervention, significantly improving the quality and usability of multimodal data.

[0106] In addition to restoring the descriptive text fragment when the semantically complex type is a complex description, the descriptive text fragment can also be corrected when the semantically complex type is a simple description. The methods for correcting the descriptive text fragment are as follows: For example, when the semantically complex type is a simple description, the current visual content corresponding to the descriptive text fragment is selected from the visual content; a semantic comparison model is used to perform semantic comparison between the descriptive text fragment and the current visual content to obtain the semantic comparison result; at least one descriptive text fragment with a different semantic comparison result is selected from the descriptive text fragments to obtain the descriptive text fragment to be updated; the descriptive text fragment to be updated is updated according to the current visual content to update the original data.

[0107] When a descriptive text fragment is judged to have a simple and direct semantic meaning, relevant current visual content can be extracted from the overall visual information based on the information conveyed by the text fragment. Because simple descriptions are relatively concise and clear, their descriptions of visual content often directly point to one or a few prominent objects, actions, or attributes. For example, the simple description "a bird is flying" allows for a more direct search for parts containing a bird in flight when filtering visual content (which could be an image or video). Compared to complex descriptions, simple descriptions do not require handling excessive details and complex logical relationships, making the filtering process relatively direct. However, it is still necessary to accurately grasp the key information conveyed by the text to ensure that the filtered current visual content accurately reflects the text description.

[0108] Semantic contrastive models are used to measure and compare the degree of semantic similarity or difference between different information sources. Through semantic parsing of text and feature extraction and semantic mapping of visual content, the model can quantify the degree of semantic matching between the two. For example, when determining whether a text describing an image semantically matches the image, the semantic contrastive model can analyze the descriptions of objects, scenes, attributes, etc. in the text and compare them with the semantic information corresponding to the visual features extracted from the image, thereby determining whether the text and visual content are consistent.

[0109] For example, consider a text snippet describing a "blue cup on a table," where the current visual content is an image of a table. A deep learning-based semantic contrast model first converts the text into word vectors, then encodes the text using a pre-trained language model to obtain semantic vectors. For the image, a convolutional neural network extracts visual features, which are then converted into semantic vectors through a fully connected layer. The text and image semantic vectors are then input into a contrast module, and cosine similarity is used to calculate their similarity, yielding the semantic contrast result. If the cosine similarity value is close to 1, it indicates a high semantic similarity between the text and image, meaning there is indeed a blue cup on the table. If the similarity value is low, such as below a preset threshold (e.g., 0.6), it indicates a semantic difference, suggesting the cup in the image may not be blue, or the cup may not even exist, requiring further verification of the text's accuracy or a re-evaluation of the image content. Besides cosine similarity, other metrics, such as KL divergence, can be used to measure the degree of difference between the text and image semantic vectors to obtain more accurate semantic contrast results.

[0110] This process involves semantically comparing text fragments describing visual content with the corresponding visual content, and then selecting semantically different text fragments as those to be updated. This allows for precise identification of problems in the data. For example, if the semantics of text fragments describing video content are inconsistent with those presented in the video frames, these fragments may mislead the model into learning incorrect cross-modal relationships. By selecting and correcting these text fragments, the quality of the training data can be improved, enabling the model to accurately learn the correct association between text and visual content, thereby enhancing the performance and accuracy of multimodal large-scale models in practical applications.

[0111] For example, the original training data might contain a text snippet describing an image as "There is a black cat sleeping in the picture," but the current visual content shows a white cat playing. This text snippet is the one that needs updating. By updating it based on the current visual content, the text snippet is changed to "There is a white cat playing in the picture," thus updating the original training data. There are several ways to update this text snippet. For instance, it can be done manually by modifying the text snippet based on the visual content; or it can be automated using natural language processing techniques. For example, a pre-trained language model can combine the feature information of the current visual content with the text snippet to be updated, allowing the model to generate a more accurate description that automatically replaces the original text snippet.

[0112] In the case of simple descriptions with complex semantics, the following steps are taken: First, the current visual content corresponding to the descriptive text fragment is selected from the visual content. This accurately locates the target visual region corresponding to the text semantics, eliminates interference from irrelevant visual information, and provides a purely objective basis for the consistency verification of simple descriptions. Then, a semantic comparison model is used to directly compare the descriptive text fragment with the current visual content. This simplifies the verification process, improves processing efficiency, and quickly obtains accurate semantic comparison results. Next, the descriptive text fragment to be updated that has a different semantic comparison result is selected from the descriptive text fragment. This directly locates noisy text in the simple description without the need for a complex mask restoration process, reducing verification costs. Finally, the descriptive text fragment to be updated is updated based on the current visual content to correct the original data. This forms the detection and correction in the simple description scenario. Text correction is completed based on real visual evidence, avoiding subjective errors and model illusions, and effectively improving the cleaning efficiency and quality of training data for simple descriptions.

[0113] After the data is updated, a notification can be displayed regarding the updated content. The notification can be displayed in the following ways: For example, the text fragment to be updated is compared with the corresponding restored text fragment; based on the comparison results, at least one text word to be updated is identified in the text fragment to be updated, and the restored text word corresponding to the text word to be updated is identified in the restored text fragment; based on the text word to be updated and the restored text word, update prompt information corresponding to the updated data is generated.

[0114] The original descriptive text fragment, to be updated, is the part whose semantics are inconsistent with the visual content. The restored descriptive text fragment, on the other hand, is generated based on the visual content and is theoretically more consistent with reality. By comparing the two, the differences between the original description and the description restored based on visual information can be clearly seen, such as word substitutions and semantic adjustments.

[0115] Among them, the text words to be updated refer to words in the descriptive text fragments that do not match the semantics conveyed by the visual content. When a semantic difference is found between the descriptive text fragments and the visual content, it is necessary to further analyze the specific words in the text fragments to find out those words that cause this difference, namely the text words to be updated.

[0116] Among them, the restored text words are appropriate words to replace the text words to be updated that do not conform to the visual semantics based on the visual content. After the text words to be updated are identified, the restored text words are generated based on the understanding and analysis of the visual content to correct the semantic deviation of the original text.

[0117] For example, the text fragment to be updated is "There is a round cup on the table, and the cup is black," while the restored text fragment is "There is a square cup on the table, and the cup is white." By comparison, we find differences between "round" and "square," and between "black" and "white." Therefore, in the text fragment to be updated, "round" and "black" are the words to be updated; in the restored text fragment, "square" and "white" are the corresponding restored text words. Identifying these word pairs can be done using rule-based comparison methods. For example, extract words describing the shape, color, and other attributes of objects from the text, and then compare whether the words in the same attribute positions in the two text fragments are consistent. Those that are inconsistent are the words to be updated and their corresponding restored text words. Alternatively, we can use word vector technology from natural language processing to calculate the similarity of word vectors in the two text fragments. Word pairs with low similarity are likely the words to be updated and their restored text words.

[0118] By generating update prompts based on the text words to be updated and the restored text words, the differences between the text description and the visual content can be clearly shown, along with how to correct them. For example, when training a multimodal model for image description generation, if there are discrepancies between the text descriptions in the training data and the image content, the generated update prompts can guide operators to accurately adjust the training data, enabling the model to learn accurate cross-modal relationships. This helps improve the accuracy and reliability of the training data, thereby enhancing the performance of the multimodal model in practical applications, such as generating more accurate text descriptions that match the image content or performing more precise image-text matching.

[0119] For example, the descriptive text fragment to be updated is "The flowers in the painting are red and square in shape," where "red" and "square" are the words to be updated. The restored descriptive text fragment is "The flowers in the painting are yellow and round in shape," where "yellow" and "round" are the restored words. Based on these words, an update prompt can be generated as: "Update 'red' to 'yellow', and 'square' to 'round'." There are several ways to generate update prompts. Besides this simple and direct word replacement prompt, it can also be presented in a structured form, such as using JSON format: {"Text word to be updated 1": "red", "Restored text word 1": "yellow", "Text word to be updated 2": "square", "Restored text word 2": "round"}. This structured information is easier for computer systems to process and parse, and can be more efficiently identified and applied in automated data update processes. In addition, update prompts can include contextual information, such as "update 'red' to 'yellow' in the description of flower color; update 'square' to 'round' in the description of flower shape," to help operators better understand the specific location and content of the update. Furthermore, update prompts can include highlighting markers; for example, based on the positions of the text word to be updated and the restored text word, the text word to be updated can be highlighted in the text segment to be updated, and the restored text word can be highlighted in the text segment to be restored.

[0120] The process involves comparing the text fragment to be updated with its corresponding restored text fragment to pinpoint the exact location of the text error, rather than simply knowing that the fragment contains an error, thus enabling fine-grained error tracing. Based on the comparison results, the system identifies the text words to be updated in the text fragment to be updated and finds the corresponding restored text words in the restored text fragment, clearly establishing a one-to-one correspondence between the erroneous and correct text, ensuring the accuracy and relevance of error location. The system also generates update prompts based on these two types of text words, visually marking the erroneous text words in the data and their corresponding corrections, allowing relevant personnel to quickly grasp the location, type, and correction results of data errors, facilitating data quality inspection, verification, and iterative optimization.

[0121] As can be seen from the above, the embodiments of this application decompose the original descriptive text into descriptive text fragments corresponding to objects in the visual content, realizing the transformation from the entire text to fine-grained semantic units, avoiding the problems of easily confused attributes and inability to locate specific erroneous text in the coarse-grained comparison of the existing overall scoring method; in addition, the semantic complexity of the descriptive text fragments is detected, and the descriptive text fragments are masked based on the semantic detection results, and the text restoration of the processed descriptive text fragments is completed based on the visual content as an objective fact, and the text content is reconstructed based on the real visual features, eliminating the semantic comparison gap and the subjectivity and instability defects of manual review; furthermore, by determining the semantic similarity between the restored descriptive text fragments and the original descriptive text fragments, noisy text that does not match the visual content can be accurately located, and the original data can be updated and corrected according to the similarity results, thereby accurately identifying noise in the data and completing targeted correction, effectively improving the quality of the data.

[0122] The method described in the above embodiments will be further described in detail below.

[0123] In this embodiment, the data processing device will be specifically integrated into an electronic device, with the electronic device serving as a terminal, as an example for explanation.

[0124] Figure 2a This is another schematic flowchart of the data processing method provided in the embodiments of this application. For example... Figure 2a As shown, a data processing method has the following specific steps: 201. The terminal splits the original description text to obtain description text fragments.

[0125] For example, the terminal processes raw training data containing visual content (image / video frame sequences) and corresponding original descriptive text. A decomposition module performs structured text splitting: the original descriptive text is input into a large model equipped with splitting rule prompts. Following the subject-verb-object logic and object attribution relationships, the model identifies and extracts the main body within the visual content as the object, along with the corresponding descriptive text fragment for each object, forming a fine-grained combination of main body and descriptive text fragment. The descriptive text fragment is the smallest verifiable logical unit, presented as a triple of "entity-attribute-relationship." One original descriptive text can be split into one or more main bodies, and one main body can correspond to one or more descriptive text fragments. Transforming the entire text into multiple independent semantic units avoids interference between different semantic parts, shifting from coarse-grained overall text to fine-grained local semantics, providing precise verification units for subsequent complexity grading and consistency checks.

[0126] For example, taking an image or video frame containing a person's legs, floor, and shoes / socks as visual content, the corresponding original descriptive text is: "The image shows the legs of a person standing on a brown wooden floor with natural texture. This person has fair skin and pinkish skin on her legs. She is wearing a set of pure white knee-high socks and a pair of beige low-top casual shoes." After inputting this original descriptive text into a large model equipped with segmentation rule prompts, it is segmented according to the subject-verb-object logic and object attribution relationship of the text. First, it identifies the two subjects in the visual content: the image and the person. The descriptive text fragment corresponding to the subject "image" is "shows the legs of a person," and the descriptive text fragments corresponding to the subject "person" are "standing on a brown wooden floor with natural texture," "fair skin," "wearing a set of pure white knee-high socks," and "wearing a pair of beige low-top casual shoes." This completes the transformation of the entire text into fine-grained semantic units, resulting in multiple independently verifiable descriptive text fragments.

[0127] 202. The terminal performs semantic detection on the descriptive text fragments to obtain the semantic complexity.

[0128] For example, the semantic complexity detection module first performs semantic recognition on the descriptive text fragments to extract their semantic logical structure information. Then, based on the semantic logical structure, it locates the associated text words (such as attribute texts like color, action, location, quantity, etc.) related to the described object's attributes. Finally, it combines three indicators—the number of text words in the descriptive text fragment, the number of associated text words, and the depth of semantic logical nesting—to comprehensively determine the semantic complexity of the text, classifying the descriptive text fragments into simple descriptions (corresponding to low and medium complexity) and complex descriptions (corresponding to high complexity). Through objective and standardized quantitative detection, it achieves accurate classification of descriptive text fragments, providing a basis for subsequent differentiated verification strategies: direct semantic comparison for simple descriptions and mask reconstruction verification for complex descriptions. This avoids redundant processing of simple text and detection failures due to judgment bias in complex text, improving overall detection efficiency and accuracy.

[0129] For example, taking all the descriptive text fragments corresponding to the main subject "character" obtained from the above split as the processing object, we first perform semantic recognition on each fragment, extract semantic logical structure information, and locate the related text words of the descriptive object's attributes; then, we combine the number of text words, the number of related text words, and the depth of logical nesting to comprehensively determine the complexity. Among them, the descriptive text fragment "standing on a brown wooden floor with natural texture" is judged as high complexity because it has many modifiers and richer logical information; the descriptive text fragment "fair skin" is short and has no complex logic, so it is judged as low complexity; the descriptive text fragment "wearing a set of pure white knee-high socks" has some attribute description but simple logic, so it is judged as medium complexity; finally, the low-complexity and medium-complexity fragments are classified as simple descriptions, and the high-complexity fragments are classified as complex descriptions, completing the standardized semantic complexity classification.

[0130] 203. The terminal performs semantic comparisons for both simple and complex descriptions to obtain semantic comparison results.

[0131] For example, based on the semantic detection results, a differentiated verification logic is used for simple and complex descriptions, and multimodal semantic alignment detection is completed by relying on the semantic similarity discrimination module.

[0132] The semantic comparison of simple descriptions adopts a direct verification mode: for simple descriptions of low and medium complexity, the corresponding current visual content is first filtered out in the visual content through visual positioning (the current visual content may include image pixel areas / video keyframes), and then the semantic comparison model is used to directly compare the description text fragment with the current visual content to quickly determine whether the two are consistent and filter out the description text fragments that do not match the semantics. The process is simple and efficient.

[0133] The semantic comparison of complex descriptions adopts a reconstruction verification mode: For highly complex descriptions, the position and corresponding attribute type of related text words are first located through the mask module, and a text mask of the same attribute type is added to obtain the processed description text fragment; then, the masked part is predicted and restored based on the current visual content. If the recognition accuracy of the text restoration model is insufficient, visual tools can be called to obtain reference information of attributes such as color, orientation, and quantity to assist in the prediction; finally, the semantic similarity between the restored description text fragment and the original description text fragment is calculated, and the similarity result is used to determine whether the text description matches the visual content.

[0134] By employing hierarchical verification, mask reconstruction, and visual tool assistance, this approach addresses the issues of attribute confusion in existing overall scoring methods and the semantic gap and inefficiency of VQA methods, achieving fine-grained and unambiguous consistency detection.

[0135] For example, for two simple descriptive fragments, the corresponding areas are first filtered out in the visual content through visual localization: for "fair skin", the skin area of ​​the person's legs is located, and for "wearing a set of pure white knee-high socks", the sock area of ​​the legs is located; then the two descriptive text fragments and the corresponding current visual content are directly input into the multimodal large model for semantic comparison. The model determines that the leg skin is light-colored, which matches the description of "fair skin", and the socks are white, knee-high style, and are a set. Both fragments are semantically consistent with the visual content.

[0136] For example, for the complex description "standing on a brown wooden floor with natural texture", the key attributes of color and material are first masked using a masking module, preserving the text context. The resulting masked text is "standing on a [color mask] floor with natural texture [material mask]". Then, the floor area in the visual content is selected as the current visual content through visual localization. The masked text and the floor area are input into a large model, which can call color recognition and material recognition tools to assist in prediction. Finally, the text is restored to "standing on a brown wooden floor with natural texture". Finally, the semantic similarity between the restored text and the original description text fragment is calculated, and it is determined that the two are highly matched, and the original description is consistent with the visual content.

[0137] 204. The terminal updates the training data based on the semantic comparison results.

[0138] For example, firstly, noisy text is filtered based on semantic comparison results: complex descriptive fragments with semantic similarity not exceeding a preset threshold and simple descriptive fragments with inconsistent semantic comparison results are identified as descriptive text fragments to be updated. Secondly, data replacement and correction are performed: in the original training data, the descriptive text fragments to be updated are replaced with accurate text obtained based on visual content reconstruction, generating updated training data. Text correction is performed based on visual content as an objective criterion, eliminating noise such as attribute errors, hallucinatory descriptions, and temporal disorder in the original data, overcoming the shortcomings of non-automatic correction, and significantly improving the quality and usability of multimodal training data.

[0139] For example, if a complex descriptive fragment in the original text contains an error, such as "standing on a white tile floor with natural textures," the correct text after semantic comparison is "standing on a brown wooden floor with natural textures." Since the semantic similarity between the two texts does not reach a preset threshold, the erroneous fragment is identified as the descriptive text fragment to be updated. In the original training data, this erroneous descriptive text fragment is replaced with the correct text, completing the update of the original training data and obtaining high-quality updated training data.

[0140] 205. The terminal generates update prompt information corresponding to the updated training data.

[0141] For example, taking the updated training data after text replacement as the processing object, firstly, the text fragments in the original training data that were judged to be erroneous and needing to be updated are compared semantically, segment by segment and word by word, with the restored text fragments that replaced those fragments. Secondly, based on the comparison results, at least one text word that does not match the visual content is accurately located in the text fragments to be updated, and at the same time, the restored text word that matches the visual reality is matched in the restored text fragments. Finally, based on the correspondence between the text words to be updated and the restored text words, update prompt information containing the error location, error text content, correct text content, and error attribute type is generated, and the error text words in the original description text are marked, highlighted, or tagged, allowing developers and operations personnel to quickly grasp the error details of the training data and correct the content.

[0142] For example, let's illustrate this with the visual content of a person, floor, shoes, and socks and their corresponding descriptive text: After updating the training data, the original descriptive text containing the incorrect text to be updated is "standing on a white tile floor with natural textures." The restored descriptive text containing this text is "standing on a brown wooden floor with natural textures." First, a semantic comparison is performed between these two text fragments, identifying two errors in the text to be updated: "white" and "tiles." Then, in the restored descriptive text, the restored text corresponding to these two errors is matched: "brown" and "wood." "White" is a color attribute error, while "tiles" is a material attribute error. Finally, based on the correspondence between the incorrect and correct text, an update prompt is generated. This prompt clearly indicates the location of the error in the descriptive text, with the incorrect text being "white" and "tiles," and the correct text being "brown" and "wood." The error types are color attribute error and material attribute error. Simultaneously, the two incorrect text words "white" and "tiles" are highlighted in the original descriptive text, allowing developers to visually see the location of the errors and the correction results.

[0143] As shown above, this solution sequentially performs fine-grained decomposition of the original description text, semantic complexity detection of description text fragments, differential semantic comparison between simple and complex descriptions, automated data updates, and generation of update prompts. Relying on large models and technologies such as visual localization, mask reconstruction, and tool-assisted prediction, it first decomposes the entire unstructured description text into independent and verifiable fine-grained semantic units, effectively avoiding interference between different semantic parts and achieving accurate error localization. Then, it differentiates the verification strategy between direct verification and reconstruction verification of different description texts by hierarchically classifying semantic complexity. This simplifies the verification process for simple descriptions, improves processing efficiency, preserves the text context framework through attribute masks, and combines visual tools to assist in reconstruction. Originally, this method solved the problems of attribute confusion, semantic gap, and inefficiency in the overall scoring method, achieving fine-grained and unambiguous consistency detection of multimodal content. Subsequently, based on the detection results, it automatically replaced erroneous text fragments with visual content as an objective basis, thoroughly eliminating noise such as attribute errors, illusory descriptions, and temporal disorder in the data. Finally, by comparing the erroneous text with the corrected text word by word, it generated update prompts with highlighted annotations, completing accurate error tracing and visual prompts, forming a closed-loop process of noise detection, automatic correction, and error prompts. This significantly improves the accuracy, reliability, and usability of multimodal data, providing high-quality data support for large multimodal models and effectively avoiding problems such as hallucinations and attribute binding errors caused by noisy data.

[0144] Figure 2b This is a schematic diagram of the terminal architecture provided in an embodiment of this application. For example... Figure 2bAs shown, the terminal architecture includes four modules: a decomposition module, a semantic complexity detection module, a masking module, and a semantic similarity discrimination module. The decomposition module receives the original descriptive text corresponding to the visual content and identifies descriptive text fragments corresponding to at least one object in the visual content within the original descriptive text. The semantic complexity detection module performs semantic detection on the descriptive text fragments, obtaining semantic detection results that indicate the semantic complexity of the descriptive text fragments. The masking module performs masking processing on the descriptive text fragments based on the semantic detection results. The semantic similarity discrimination module performs text reconstruction on the processed descriptive text fragments based on the visual content, determines the semantic similarity between the reconstructed descriptive text fragments and the original descriptive text fragments, and determines the consistency between the reconstructed descriptive text fragments and the original descriptive text fragments based on the semantic similarity.

[0145] The decomposition module functions by breaking down the complex original descriptive text corresponding to visual content into multiple independent semantic parts (i.e., descriptive text fragments). This avoids interference between the information in these independent semantic parts, thereby improving the accuracy and recall of the subsequent consistency determination module. During decomposition, each subject has its corresponding descriptive text fragment. The process involves inputting the original descriptive text into a large model, followed by inputting prompts with decomposition rules to the model for further decomposition. An object with numerous attribute information can be considered a single subject; alternatively, each object in the original descriptive text can be treated as a subject to obtain multiple subjects; furthermore, the original descriptive text can be decomposed into subjects and their corresponding descriptive text fragments based on its semantic logical structure, such as subject-verb-object. An original descriptive text can include one or more subjects; a subject can correspond to only one descriptive text fragment; or a subject can correspond to multiple descriptive text fragments.

[0146] Figure 2c This is a schematic diagram of the disassembly module provided in an embodiment of this application. For example... Figure 2c As shown, the decomposition module includes a large model and prompt words. The input of this module is the original descriptive text, and the output of this module is the main body and the corresponding descriptive text fragments.

[0147] For example, an input example for the disassembly module could be: The image shows the legs of a person standing on a brown wooden floor with a natural texture. The person has fair skin, with pinkish skin on their legs. They are wearing pure white knee-high socks and beige low-top casual shoes.

[0148] The input example for the decomposition module is a complete, undivided original visual content description text. This text is generated around an image or video frame containing a person's legs, floor, shoes and socks. It integrates comprehensive information from multiple objects and attributes, such as the overall presentation of the scene, the person's posture, skin color, and clothing. It is a typical unstructured whole text and is the raw data for fine-grained semantic deconstruction.

[0149] For example, the output of the disassembly module could be: [ { "subject" / / main body: "image", "desc" / / Describes a text snippet:[ "It shows the legs of a figure." ] }, { "subject" / / main body: "person", "desc": / / describes a text fragment[ Standing on a brown wooden floor with a natural texture, "Fair skin" "Wearing a set of pure white knee-high stockings", Wearing a pair of beige low-top casual shoes ] } ] The output example of the decomposition module is a structured result obtained by splitting the large model with splitting rule prompts according to subject attribution and subject-verb-object logic. The module accurately identifies the two subjects, "image" and "person", and matches the corresponding independent descriptive text fragments for each subject. It transforms the unstructured whole text into fine-grained semantic units of multiple subjects and descriptive text fragments, and each fragment is the smallest verifiable logical unit.

[0150] The semantic complexity detection module functions as follows: before sending a descriptive text fragment to a downstream large model for semantic similarity judgment, it first detects the complexity of the descriptive text fragment itself, and then selects the optimal consistency verification path based on the detection results. During complexity detection, data can be manually labeled before being processed by the model; alternatively, the descriptive text fragment can be directly input into the large model, and complexity can be detected using rules from prompt words. In complexity detection, semantic recognition is performed on the descriptive text fragment to obtain semantic logical structure information. Then, related text words are identified based on this semantic logical structure information. Finally, the semantic complexity is obtained based on metrics such as description length, the number of related text occurrences, and semantic logical structure information.

[0151] Figure 2d This is a schematic diagram illustrating the operation of the semantic complexity detection module provided in an embodiment of this application. For example... Figure 2d As shown, the semantic complexity detection module includes a large model and prompt words. The input of this module is the subject and the subject description, and the output of this module is the subject, the subject description, and the complexity.

[0152] For example, the input example for the semantic complexity detection module can be: [ { "subject" / / main body: "person", "desc" / / Describes a text snippet:[ Standing on a brown wooden floor with a natural texture, "Fair skin" "Wearing a set of pure white knee-high socks" ] } ] The input example for the semantic complexity detection module is the simplified structured data output by the decomposition module. Only the main subject "person" and its three corresponding descriptive text fragments are retained, and irrelevant subject information is removed. It is a pure text semantic unit that has been decomposed into fine granularity, and serves as the processing object for the module to predict and classify semantic complexity.

[0153] For example, the output of the semantic complexity detection module can be: [ { "subject" / / main body: "person", "desc" / / Describes a text snippet:[ { "text" / / Text: "Standing on a brown wooden floor with a natural texture", "complexity" / / Complexity: "High" }, { The text in the title is "fair skin". "complexity" / / Complexity: "Low" }, { "text" / / Text: "Wearing a set of pure white knee-high socks", "complexity" / / Complexity: "Medium" } ] } ] The output example of the semantic complexity detection module is a structured result that labels each descriptive text fragment with a corresponding complexity level after the large model evaluates it based on three indicators: description length, number of associated texts, and semantic logical structure information. The module judges long texts with multiple attributes as high complexity, short texts with a single attribute as low complexity, and descriptions with medium attributes as medium complexity, clearly distinguishing between simple and complex description types.

[0154] The masking module functions as follows: for descriptive text fragments with high semantic complexity, it performs targeted or random masking of related text words such as color, quantity, and position, while preserving the context. During masking, one or more related text words can be masked at a time.

[0155] Figure 2e This is a schematic diagram of the operation of the mask module provided in the embodiments of this application, such as... Figure 2e As shown, the masking module includes a large model and prompt words. The input of this module is the main body, a descriptive text fragment, and the complexity. The output of this module is the main body, the processed descriptive text fragment, and the complexity.

[0156] For example, the input example for the mask module can be: [ { "subject" / / main body: "person", "desc" / / Describes a text snippet:[ { "text" / / Text: "Standing on a brown wooden floor with a natural texture", "complexity" / / Complexity: "High" } ] } ] The input example for the masking module is the data corresponding to high complexity filtered by the semantic complexity detection module. It only contains the main subject "person" and its single descriptive text fragment marked as high complexity. It is the processing object for the module to perform masking operations on complex descriptions.

[0157] For example, the output of the mask module can be: [ { "subject" / / main body: "person", "desc" / / Describes a text snippet:[ { "text" / / Text: "Standing on a [color mask] with a natural texture [material mask]", "complexity" / / Complexity: "High" } ] } ] The output example of the masking module is the text result after targeted masking of the input high-complexity descriptive text according to attribute types such as color and material. The module only masks some related text words and fully preserves the original sentence's contextual semantic framework, replacing related text words with mask placeholders of the same type without destroying the text's logical structure.

[0158] The semantic similarity discrimination module functions by applying different consistency checking schemes to descriptive text fragments with varying semantic complexity. This is because large models typically provide stable and reliable outputs for simple, explicit inputs; however, when faced with complex, detailed texts containing multiple logical relationships, their attention may be distracted, or they may become misled by certain details.

[0159] Therefore, when the module determines that the input descriptive text is simple, direct, and unambiguous enough, it directly sends the descriptive text fragment corresponding to the simple description and the corresponding visual content to the downstream large model and accepts the consistency judgment result returned by it.

[0160] Therefore, when the module determines that the input descriptive text is complex, contains multiple subjects, rich actions, or may be ambiguous, it masks related text words in the corresponding descriptive text fragments, such as subjects, core actions, or important attributes. This masked text is then fed into a larger model, which is asked to complete the masked portion. The model's completed content is compared with the original masked content to determine if they are consistent or highly similar semantically. When judging whether the completed content is consistent or highly similar to the original masked content, similarity can be used, calculated using cosine similarity or vector similarity; alternatively, the large model can be used by inputting rule-based prompts and obtaining the similarity score. When predicting and completing the masked portion, for factual descriptions such as color, quantity, and location, the large model can utilize provided tools to obtain attribute reference information.

[0161] Figure 2f This is a schematic diagram of the semantic similarity discrimination module provided in the embodiments of this application, such as... Figure 2f As shown, the semantic similarity discrimination module includes a large model, prompt words, and tools. The input of this module is the subject, descriptive text fragments, complexity, and visual content. The output of this module is the semantic similarity discrimination result.

[0162] For example, the input example for the semantic similarity discrimination module can be: [ { "subject" / / main body: "person", "desc" / / Describes a text snippet:[ { "text" / / Text: "Standing on a brown wooden floor with a natural texture", "complexity" / / Complexity: "High" }, { "text" / / Text: "Fair skin", "complexity" / / Complexity: "Low" }, { "text" / / Text: "Wearing a set of pure white knee-high socks", "complexity" / / Complexity: "Medium" } ] } ] The input example for the semantic similarity discrimination module is the complete processing result of all preceding modules, including the subject "person", all descriptive text fragments of three complexity levels (high / low / medium) and corresponding visual content, covering two verification types: simple description and complex description.

[0163] For example, the output of the semantic similarity discrimination module can be: For semantically simple descriptive text fragments: [ { "subject" / / main body: "person", "desc" / / Describes a text snippet:[ { "text" / / Text: "Fair skin", "complexity_level" / / Complexity level: "low" "conclusion" / / Result: "true", / / Conforms to the condition "reason" / / Judgment Reason: "Regarding 'skin tone': Leg skin is visible in the video. Regarding 'fair skin': The skin color is indeed fair, conforming to color rules. The skin in the video appears in a light color scheme, consistent with the description of 'fair skin'." }, { "text" / / Text: "Wearing a set of pure white knee-high socks", "complexity_level" / / Complexity level: "Medium" "conclusion" / / Result: "true" "reason" / / Judgment Reasons: "Regarding 'clothing': The video clearly shows the person wearing socks. Regarding 'a set': The video shows two feet, constituting a set. Regarding 'pure white': The socks are indeed white, conforming to the principle of color consistency. Regarding 'knee-high': The socks do reach the knee. Regarding 'high socks': The socks are a high-top design, consistent with the description." } ] } ] For semantically complex descriptive text fragments: [ { "subject" / / main body: "person", "desc" / / Describes a text snippet:[ { "text" / / Text: "Standing on a [color mask] with a natural texture [material mask]", "complexity" / / Complexity level: "high" "answer" / / Complete the answer: "Standing on a brown wooden floor with natural texture", "conclusion" / / Result: "true" "reason" / / Reason for judgment: "In the video, the floor is a typical wooden structure, and its color is brown. Comparison with the original text shows a high degree of consistency between the original text and the answer; therefore, the conclusion is true." } ] } ] The semantic similarity discrimination module outputs two types of verification results: simple descriptions and complex descriptions. For simple descriptions of low and medium complexity, it directly outputs the semantic consistency judgment conclusion and detailed verification reasons. For complex descriptions of high complexity, it outputs the processed description text fragment, the restored description text fragment based on the model's visual content completion, the semantic similarity discrimination result, and the verification reasons, fully presenting the entire hierarchical verification process and accurately reflecting the matching status of each description text fragment with the visual content.

[0164] As shown above, decomposing the original descriptive text into descriptive text fragments corresponding to objects in the visual content achieves the transformation from the entire text to fine-grained semantic units, avoiding the problems of easily confused attributes and inability to locate specific erroneous text in existing overall scoring methods with coarse-grained comparison. In addition, semantic complexity detection is performed on the descriptive text fragments, and the descriptive text fragments are masked based on the semantic detection results. The text of the processed descriptive text fragments is restored based on the visual content as an objective fact, and the text content is reconstructed based on the real visual features, eliminating the semantic comparison gap and the subjectivity and instability defects of manual review. Furthermore, by determining the semantic similarity between the restored descriptive text fragments and the original descriptive text fragments, noisy text that does not match the visual content can be accurately located. Then, the original data is updated and corrected based on the similarity results, thereby accurately identifying noise in the data and completing targeted corrections, effectively improving the quality of the data.

[0165] To better implement the above methods, this application also provides a data processing device, which can be integrated into an electronic device, such as a server or terminal. The terminal may include a tablet computer, a laptop computer, and / or a personal computer.

[0166] For example, see Figure 3 , Figure 3 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application. Figure 3 As shown, the data processing device may include an acquisition unit 301, an identification unit 302, a detection unit 303, a restoration unit 304, and an update unit 305, as follows: (1) Obtain unit 301; The acquisition unit 301 is used to acquire at least one piece of raw data, which includes visual content and the original descriptive text corresponding to the visual content.

[0167] (2) Identification unit 302; The recognition unit 302 is used to identify a descriptive text fragment corresponding to at least one object in the visual content in the original descriptive text.

[0168] For example, the identification unit 302 can be specifically used for: Semantic recognition is performed on the original descriptive text to obtain its semantic logical structure information; based on the semantic logical structure information of the original descriptive text, at least one object in the visual content is identified in the original descriptive text; the text content corresponding to the object is extracted from the original descriptive text to obtain a descriptive text fragment.

[0169] (3) Detection unit 303; The detection unit 303 is used to perform semantic recognition on the descriptive text fragment to obtain the semantic logical structure information of the descriptive text fragment. It is also used to identify at least one associated text word in the descriptive text fragment based on the semantic logical structure information. The associated text word is a text word used to describe the attribute information of the object. It is also used to detect the semantic complexity of the descriptive text fragment based on the number of text words in the descriptive text fragment, the number of associated text words, and the semantic logical structure information, and obtain a semantic detection result. The semantic detection result indicates the semantic complexity of the descriptive text fragment.

[0170] (4) Reduction unit 304; The restoration unit 304 is used to perform masking processing on the descriptive text fragment based on the semantic detection results, and to restore the processed descriptive text fragment based on the visual content.

[0171] For example, the restoration unit 304 can be specifically used for: Based on the semantic detection results, the semantic complexity type of the descriptive text fragment is determined. The semantic complexity type includes either simple description or complex description. When the semantic complexity type is complex description, the text word position of the associated text word is identified in the descriptive text fragment, and the attribute type corresponding to the associated text word is determined. Based on the text word position, a text mask corresponding to the attribute type is added to the descriptive text fragment to obtain at least one processed descriptive text fragment.

[0172] For example, the restoration unit 304 can also be used for: Based on the content type of the visual content, the current visual content corresponding to the descriptive text fragment is selected from the visual content; based on the current visual content, text prediction is performed on the text mask in the processed descriptive text fragment to obtain the predicted text words corresponding to the text mask; the text mask is replaced with the predicted text words in the processed descriptive text fragment to obtain the restored descriptive text fragment.

[0173] For example, the restoration unit 304 can also be used for: Based on the current visual content, a text reconstruction model is used to predict the text mask in the processed descriptive text fragment. When the text reconstruction model cannot recognize the current visual content, or the accuracy of the text prediction result does not reach the preset accuracy threshold, attribute reference information of the attribute type corresponding to the text mask is obtained. Based on the attribute reference information and the current visual content, the text reconstruction model is used to predict the text mask in the processed descriptive text fragment, and the predicted text words corresponding to the text mask are obtained.

[0174] For example, the restoration unit 304 can also be used for: When the content type is video content, at least one video frame corresponding to the descriptive text fragment is selected from the visual content to obtain the current visual content; when the content type is image content, the pixel area corresponding to the descriptive text fragment is selected from the visual content to obtain the current visual content.

[0175] For example, the data processing apparatus further includes a comparison unit, which can specifically be used for: This method is used to filter out the current visual content corresponding to the descriptive text fragment when the semantically complex type is a simple description; it uses a semantic comparison model to perform semantic comparison between the descriptive text fragment and the current visual content to obtain the semantic comparison result; it filters out at least one descriptive text fragment whose semantic comparison result is different from the descriptive text fragment to obtain the descriptive text fragment to be updated; and it updates the descriptive text fragment to be updated according to the current visual content to update the original data.

[0176] (5) Update unit 305; The update unit 305 is used to determine the semantic similarity between the restored descriptive text fragment and the original descriptive text fragment, and update the original data based on the semantic similarity.

[0177] For example, update unit 305 can be used specifically for: In the descriptive text fragments, at least one descriptive text fragment with a semantic similarity not exceeding a preset similarity threshold is selected to obtain the descriptive text fragment to be updated; in the original data, the descriptive text fragment to be updated is replaced with the corresponding restored descriptive text fragment to obtain the updated data.

[0178] For example, the data processing apparatus also includes a prompting unit, which can specifically be used for: The text fragment to be updated is compared with the corresponding restored text fragment. Based on the comparison results, at least one text word to be updated is identified in the text fragment to be updated, and the restored text word corresponding to the text word to be updated is identified in the restored text fragment. Based on the text word to be updated and the restored text word, update prompt information corresponding to the updated data is generated.

[0179] Furthermore, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the data processing method provided in embodiments of this application.

[0180] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0181] As can be seen from the above, the identification unit 302 in this embodiment decomposes the original descriptive text into descriptive text fragments corresponding to objects in the visual content, realizing the transformation from the entire text to fine-grained semantic units, avoiding the problems of easily confused attributes and inability to locate specific erroneous text in the coarse-grained comparison of the existing overall scoring method; in addition, the detection unit 303 performs semantic complexity detection on the descriptive text fragments, and the restoration unit 304 performs masking processing on the descriptive text fragments based on the semantic detection results, and completes the text restoration of the processed descriptive text fragments based on the visual content as an objective fact, reconstructing the text content based on the real visual features, eliminating the semantic comparison gap and the subjectivity and instability defects of manual review; in addition, the update unit 305 can accurately locate noisy text that does not match the visual content by determining the semantic similarity between the restored descriptive text fragments and the original descriptive text fragments, and then update and correct the original data according to the similarity results, thereby accurately identifying noise in the data and completing targeted correction, effectively improving the quality of the data.

[0182] This application also provides an electronic device. Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 4 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically: The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402, to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0183] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0184] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0185] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0186] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows: The process involves: acquiring at least one set of raw data, including visual content and corresponding raw descriptive text; identifying descriptive text fragments corresponding to at least one object in the visual content from the raw descriptive text; performing semantic recognition on the descriptive text fragments to obtain their semantic logical structure information; identifying at least one associated text word in the descriptive text fragments based on the semantic logical structure information, where the associated text word is a text word used to describe the attribute information of the object; detecting the semantic complexity of the descriptive text fragments based on the number of text words, the number of associated text words, and the semantic logical structure information, and obtaining a semantic detection result indicating the semantic complexity of the descriptive text fragments; masking the descriptive text fragments based on the semantic detection result, and restoring the processed descriptive text fragments based on the visual content; determining the semantic similarity between the restored descriptive text fragments and the original data based on the semantic similarity.

[0187] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0188] As can be seen from the above, the embodiments of this application decompose the original descriptive text into descriptive text fragments corresponding to objects in the visual content, realizing the transformation from the entire text to fine-grained semantic units, avoiding the problems of easily confused attributes and inability to locate specific erroneous text in the coarse-grained comparison of the existing overall scoring method; in addition, the semantic complexity of the descriptive text fragments is detected, and the descriptive text fragments are masked based on the semantic detection results, and the text restoration of the processed descriptive text fragments is completed based on the visual content as an objective fact, and the text content is reconstructed based on the real visual features, eliminating the semantic comparison gap and the subjectivity and instability defects of manual review; furthermore, by determining the semantic similarity between the restored descriptive text fragments and the original descriptive text fragments, noisy text that does not match the visual content can be accurately located, and the original data can be updated and corrected according to the similarity results, thereby accurately identifying noise in the data and completing targeted correction, effectively improving the quality of the data.

[0189] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0190] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the data processing methods provided in embodiments of this application. For example, the instructions can execute the following steps: The process involves: acquiring at least one set of raw data, including visual content and corresponding raw descriptive text; identifying descriptive text fragments corresponding to at least one object in the visual content from the raw descriptive text; performing semantic recognition on the descriptive text fragments to obtain their semantic logical structure information; identifying at least one associated text word in the descriptive text fragments based on the semantic logical structure information, where the associated text word is a text word used to describe the attribute information of the object; detecting the semantic complexity of the descriptive text fragments based on the number of text words, the number of associated text words, and the semantic logical structure information, and obtaining a semantic detection result indicating the semantic complexity of the descriptive text fragments; masking the descriptive text fragments based on the semantic detection result, and restoring the processed descriptive text fragments based on the visual content; determining the semantic similarity between the restored descriptive text fragments and the original data based on the semantic similarity.

[0191] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0192] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0193] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the data processing methods provided in the embodiments of this application, the beneficial effects that any of the data processing methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0194] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in various alternative implementations of the data processing method described above.

[0195] The data processing method, apparatus, device, product, and storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data processing method, characterized in that, include: Acquire at least one piece of raw data, the raw data including visual content and the original descriptive text corresponding to the visual content; Identify the descriptive text fragment corresponding to at least one object in the visual content from the original descriptive text; Semantic recognition is performed on the descriptive text fragment to obtain the semantic logical structure information of the descriptive text fragment; Based on the semantic logic structure information, at least one associated text word is identified in the descriptive text fragment, and the associated text word is a text word used to describe the attribute information of the object; Based on the number of text words in the descriptive text fragment, the number of associated text words, and the semantic logic structure information, the semantic complexity of the descriptive text fragment is detected to obtain a semantic detection result, which indicates the semantic complexity of the descriptive text fragment. Based on the semantic detection results, the descriptive text fragment is masked, and based on the visual content, the processed descriptive text fragment is restored. Determine the semantic similarity between the restored descriptive text fragment and the original descriptive text fragment, and update the original data based on the semantic similarity.

2. The data processing method according to claim 1, characterized in that, The masking process for the descriptive text fragment based on the semantic detection result includes: Based on the semantic detection results, the semantic complexity type of the descriptive text fragment is determined, and the semantic complexity type includes one of simple description and complex description; When the semantic complexity type is the complex description, the text word position of the associated text word is identified in the description text fragment, and the attribute type corresponding to the associated text word is determined; Based on the position of the text word, a text mask corresponding to the attribute type is added to each of the descriptive text fragments to obtain at least one of the processed descriptive text fragments.

3. The data processing method according to claim 2, characterized in that, The step of restoring the processed descriptive text fragment based on the visual content includes: Based on the content type of the visual content, the current visual content corresponding to the descriptive text fragment is selected from the visual content; Based on the current visual content, text prediction is performed on the text mask in the processed descriptive text fragment to obtain the predicted text word corresponding to the text mask; The text mask is replaced with the predicted text word in the processed descriptive text fragment to obtain the restored descriptive text fragment.

4. The data processing method according to claim 3, characterized in that, The step of predicting text based on the text mask in the processed descriptive text fragment, based on the current visual content, to obtain the predicted text words corresponding to the text mask, includes: Based on the current visual content, a text restoration model is used to predict the text mask in the processed descriptive text fragment. When the text restoration model cannot recognize the current visual content, or when the accuracy of the text prediction result does not reach the preset accuracy threshold, the attribute reference information of the attribute type corresponding to the text mask is obtained. Based on the attribute reference information and the current visual content, the text restoration model is used to predict the text mask in the processed descriptive text fragment, thereby obtaining the predicted text word corresponding to the text mask.

5. The data processing method according to claim 3, characterized in that, The step of filtering out the current visual content corresponding to the descriptive text fragment from the visual content based on the content type of the visual content includes: When the content type is video content, at least one video frame corresponding to the descriptive text fragment is selected from the visual content to obtain the current visual content. When the content type is image content, the pixel region corresponding to the descriptive text fragment is selected from the visual content to obtain the current visual content.

6. The data processing method according to claim 2, characterized in that, Also includes: When the semantically complex type is the simple description, the current visual content corresponding to the descriptive text fragment is filtered out from the visual content; A semantic comparison model is used to semantically compare the descriptive text fragment with the current visual content to obtain the semantic comparison result. Filter out at least one descriptive text fragment whose semantic comparison result is different from the descriptive text fragments to obtain the descriptive text fragment to be updated; Based on the current visual content, the descriptive text fragment to be updated is updated to update the original data.

7. The data processing method according to claim 1, characterized in that, The step of updating the original data based on the semantic similarity includes: At least one descriptive text fragment whose semantic similarity does not exceed a preset similarity threshold is selected from the descriptive text fragments to obtain the descriptive text fragment to be updated; In the original data, the text fragment to be updated is replaced with the corresponding restored text fragment to obtain the updated data.

8. The data processing method according to claim 7, characterized in that, Also includes: Compare the text fragment to be updated with the corresponding restored text fragment; Based on the comparison results, at least one text word to be updated is identified in the text fragment to be updated, and the restored text word corresponding to the text word to be updated is identified in the restored text fragment. Based on the text words to be updated and the restored text words, an update prompt message corresponding to the updated data is generated.

9. The data processing method according to claim 1, characterized in that, The step of identifying the descriptive text fragment corresponding to at least one object in the visual content within the original descriptive text includes: Semantic recognition is performed on the original description text to obtain the semantic logical structure information of the original description text; Based on the semantic logic structure information of the original description text, at least one object in the visual content is identified in the original description text; The text content corresponding to the object is extracted from the original description text to obtain the description text fragment.

10. A data processing apparatus, characterized in that, include: An acquisition unit is used to acquire at least one piece of raw data, the raw data including visual content and the original descriptive text corresponding to the visual content; A recognition unit is configured to identify, in the original description text, a description text fragment corresponding to at least one object in the visual content; The detection unit is configured to perform semantic recognition on the descriptive text fragment to obtain semantic logical structure information of the descriptive text fragment, and is further configured to identify at least one associated text word in the descriptive text fragment based on the semantic logical structure information, wherein the associated text word is a text word used to describe the attribute information of the object, and is further configured to detect the semantic complexity of the descriptive text fragment based on the number of text words in the descriptive text fragment, the number of associated text words, and the semantic logical structure information, and obtain a semantic detection result, wherein the semantic detection result indicates the semantic complexity of the descriptive text fragment; The restoration unit is used to perform masking processing on the descriptive text fragment based on the semantic detection result, and to restore the processed descriptive text fragment based on the visual content. An update unit is used to determine the semantic similarity between the restored descriptive text fragment and the original descriptive text fragment, and to update the original data based on the semantic similarity.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing an application program, and the processor running the application program within the memory to perform the steps of the data processing method according to any one of claims 1 to 9.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the data processing method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the data processing method according to any one of claims 1 to 9.

Images