Method for determining object classification, related device and computer program product

Abstract

The invention provides a method for determining object classification, a related device and a computer program product, and the method comprises the steps: processing first modal data of a to-be-classified object through a data classification model, and obtaining a first classification result of the first modal data of an application embodiment and a first classification confidence coefficient for the first classification result of the application embodiment, the object to be detected in the application embodiment is composed of first modal data and second modal data, and the data volume of the first modal data in the application embodiment is smaller than that of the second modal data in the application embodiment; and in response to the first classification result of the application embodiment indicating that the first modal data of the application embodiment is the first data type and the first classification confidence of the application embodiment is greater than or equal to a first confidence threshold, determining the to-be-classified object of the application embodiment as the first data type of the application embodiment. Therefore, the data classification efficiency of the multi-modal data can be improved on the basis of guaranteeing the data classification quality of the multi-modal data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for determining object classification, an electronic device, a computer-readable medium, and a computer program product. Background Art

[0002] With the development of society, computer technology is also improving day by day. The Internet can provide a platform for users to communicate, so that users can communicate, exchange and share content with other users by posting videos and information on the Internet. For example, users can achieve "online communication" with other users by posting information.

[0003] In this context, in order to facilitate users to obtain content and communicate with other users, "platforms" came into being. Accordingly, the existence of "platforms" enables different users to share and communicate content without establishing any connection relationship such as friendship.

[0004] In addition, in order to better serve users, the platform will choose to provide users with and allow users to upload content of different types and modes. For example, users are allowed to upload user content in various forms and modes, such as text, images, audio, and video. In other words, users are allowed to provide multimodal content. Therefore, when users provide content of different types and modes, how to ensure the safety, friendliness, and health of communication on the platform in a quality and effective manner and avoid the mis-spreading of unfriendly and unhealthy information is worthy of attention and urgent need. Summary of the invention

[0005] Multiple aspects of the present application provide a method, device, electronic device, computer-readable storage medium and computer program product for determining object classification, which can improve the efficiency of data classification of multimodal data while ensuring the quality of data classification of multimodal data.

[0006] In one aspect of the present application, a method for determining object classification is provided, comprising: using a data classification model to process first modal data of an object to be classified, obtaining a first classification result of the first modal data and a first classification confidence for the first classification result, wherein the object to be detected is composed of first modal data and second modal data, and the data volume of the first modal data is smaller than that of the second modal data; in response to the first classification result indicating that the first modal data is a first data type, and the first classification confidence is greater than or equal to a first confidence threshold, determining the object to be classified as the first data type.

[0007] In another aspect of the present application, there is provided an apparatus for determining object classification, comprising: a first data processing module, configured to process first modal data of an object to be classified using a data classification model, to obtain a first classification result of the first modal data and a first classification confidence for the first classification result, wherein the object to be detected is composed of first modal data and second modal data, and the data volume of the first modal data is smaller than that of the second modal data; a first classification determination module, configured to determine the object to be classified as the first data type in response to the first classification result indicating that the first modal data is a first data type, and the first classification confidence is greater than or equal to a first confidence threshold.

[0008] Another aspect of the present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for determining object classification as provided above.

[0009] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method for determining object classification as provided above.

[0010] In another aspect of the present application, a computer program product includes a computer program having computer program instructions stored thereon. When the computer program is executed by a processor, the method for determining the classification of an object as provided above can be implemented.

[0011] In the solution provided by the embodiment of the present application, the first modal data of the object to be classified is processed by using a data classification model to obtain a first classification result of the first modal data and a first classification confidence for the first classification result, wherein the object to be detected is composed of the first modal data and the second modal data, and the data volume of the first modal data is smaller than that of the second modal data; in response to the first classification result indicating that the first modal data is the first data type, and the first classification confidence is greater than or equal to the first confidence threshold, the object to be classified is determined to be the first data type. In this way, the efficiency of data classification for multimodal data can be improved while ensuring the quality of data classification for multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0013] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0014] Figure 1 A schematic diagram of a process for determining object classification provided in an embodiment of the present application;

[0015] Figure 2 A schematic diagram of another process for determining object classification provided in an embodiment of the present application;

[0016] Figure 3 A schematic diagram of an architecture of a process that can be used to determine object classification in an application scenario provided by an embodiment of the present application;

[0017] Figure 4 A schematic diagram of the structure of an apparatus for determining object classification provided in an embodiment of the present application;

[0018] Figure 5 The present invention is a schematic diagram of the structure of an electronic device suitable for implementing the solution in the embodiment of the present application.

[0019] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0022] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0023] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0024] As discussed above, when providing different types of service content, how to ensure the safety, friendliness and health of communication on the platform in a high-quality and effective manner and avoid the incorrect spread of unfriendly and unhealthy information is worthy of attention and an urgent need.

[0025] In some solutions, the choice is made to simultaneously process and classify data from different modalities belonging to the same object. This approach not only places too high demands on the computing power of the classification system, but also leads to problems such as difficulty in ensuring classification quality due to excessive data volume and cross-modality, and long waiting cycles for processing.

[0026] In this regard, an embodiment of the present application provides a method for determining object classification, which uses a data classification model to process the first modal data of an object to be classified, and obtains a first classification result of the first modal data and a first classification confidence for the first classification result, wherein the object to be detected is composed of the first modal data and the second modal data, and the data volume of the first modal data is smaller than that of the second modal data; in response to the first classification result indicating that the first modal data is a first data type, and the first classification confidence is greater than or equal to a first confidence threshold, the object to be classified is determined to be the first data type. In this way, the efficiency of data classification for multimodal data can be improved while ensuring the quality of data classification for multimodal data.

[0027] In actual scenarios, the execution subject of the method can be a user device, or a device formed by integrating a user device and a network device through a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0028] When the execution subject is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0029] Figure 1 A process 100 for determining an object classification provided by an embodiment of the present application is shown. The process 100 includes at least the following processing steps:

[0030] (Step) S101, using a data classification model to process first modal data of an object to be classified, to obtain a first classification result of the first modal data and a first classification confidence for the first classification result.

[0031] In an embodiment of the present application, after acquiring the object to be classified (for example, "multimodal content" uploaded by a user including data of at least two modalities), the executing entity (for example, the server used by the "platform") determines and separates the modal data under each modality that constitutes the object to be classified.

[0032] In practice, the "data modality" may be, for example, text, image, audio, etc. Accordingly, for example, the actual data form of the object to be classified may be text-image data, text-audio data, image-audio data, text-image-audio data, etc.

[0033] Text-Image Data: This type of data contains a combination of text and images, such as image captioning tasks where each image has an associated text description.

[0034] Text-audio data: This type of data combines text and audio information, for example, the output of speech recognition may include transcribed text as well as the original audio.

[0035] Text-video data: This type of data contains both text and video information, such as automatic annotation of video content, where the text can describe the scene or action in the video.

[0036] Image-audio data: This data combines image and audio information, such as a music video, where the image shows the musician or the music being played, and the audio is the music itself.

[0037] Text-Image-Audio Data: This type of data contains a combination of text, images, and audio, such as an online news article containing multimedia elements, which may have a text description, related pictures and videos, and an audio interview.

[0038] It should be understood that, in practice, the object to be classified may actually be composed of modal data of more modalities (eg, first modal data, second modal data, and third modal data).

[0039] Then, the execution subject can determine the size of the data volume corresponding to each modal data of the object to be classified. Then select the modal data with the smallest data volume, or data volume. For ease of understanding, when the execution subject determines that the object to be classified is composed of modal data of two modalities, the modal data with a smaller data volume can be exemplarily referred to as the first modal data, and the other, as the second modal data. That is, in the case where, for example, the object to be detected is composed of the first modal data and the second modal data, the data volume of the first modal data discussed in this article is smaller than the second modal data discussed. Accordingly, the first modal data and the second modal data are different modalities, for example, the first modal data is data under the text modality, and the second modal data is data under the image modality.

[0040] It should be understood that in some embodiments, if there are multiple modal data, the execution entity can similarly determine the first modal data, the second modal data, the third modal data, and so on (wherein it can be understood that the data volume of the first modal data is smaller than that of the second modal data, and the data volume of the second modal data is smaller than that of the third modal data).

[0041] Next, in this step, the execution subject may first select the first modal data as a processing object to preferentially process the first modal data to obtain a first classification result and a first classification confidence of the first modal data.

[0042] For example, the execution entity may determine, based on the data features of the first modal data, whether it includes the “target features” corresponding to each data classification (for example, based on the similarity comparison results between the data features and the target features, if the similarity meets the requirements, it is considered that the first modal data includes the “target features”, and the first modal data is correspondingly determined to be the data type corresponding to the “target features”).

[0043] In an embodiment of the present application, a data type may include at least a first data type and a second data type, wherein the first data type may be exemplified as one that may not be expected to be directly used or published. For example, if data or an object is classified as a first data type, the data object may not be directly used or published (for example, further verification of publishing or usage permissions may be required). For example, in some embodiments, for an object to be classified, if one of the data included therein is data of a modality that is determined to be a "first data type", then for the object to be classified, due to factors such as the relevance of its content, the object to be classified may be classified as a whole to indicate that the object to be classified as a whole "cannot be directly used or published".

[0044] Correspondingly, the second data type is a data type different from the first data type. For example, it can be considered as a data type that can be directly used and published in the current state.

[0045] Accordingly, the execution subject may use the data classification model to process the first modal data to determine the corresponding first data classification result (for example, the first data type or the second data type). In some embodiments, the data classification model may be pre-configured corresponding to the modality. For example, for image data, a ResNet, ViT or other model may be used to process and classify it; for text data, a natural language processing model may be used to process and classify it; for voice data, an automatic speech recognition (ASR) model may be used to process and classify it.

[0046] In this process, the execution subject can also use the data classification model to output the first classification confidence of the data classification model for the first classification result obtained. The first classification confidence is used to indicate the "credibility" and "reliability" of the output first classification result.

[0047] In some embodiments, the first classification confidence is obtained by processing the hidden layer vector output by the last layer decoder in the data classification model by a classification head configured in the data classification model.

[0048] Specifically, in the data classification model, a classification head can also be configured to determine the "classification loss" (or classification loss) and then determine the (first) classification confidence. The classification head can be understood as the part used to map the learned feature representation to the category prediction. The classification head can usually include one or more fully connected layers for mapping features to the category space, and an activation function, such as Softmax, for generating a probability distribution of "whether the classification result is reliable" and determining the classification confidence.

[0049] Accordingly, the data classification model can be configured to use the classification head to process the hidden layer vector output by the last layer decoder in the data classification model to generate a first classification confidence for the first classification result. Thus, the classification head configured in the data classification model can be directly used to supervise the "classification process" and refer to the "classification process", and the corresponding first classification confidence can be generated directly based on the supervision result and reference result of the "classification process", which can directly evaluate the "classification process" while simplifying the overall system structure configuration, thereby improving the reliability of the generation of classification confidence.

[0050] S102, determining whether the first classification result indicates that the first modal data is a first data type or a second data type.

[0051] If the first classification result indicates that the first modal data is of the first data type, the execution entity may choose to continue executing S103.

[0052] S103: Determine whether the first classification confidence is greater than or equal to a first confidence threshold.

[0053] In an embodiment of the present application, the first confidence threshold can be set according to the scenario and requirements. For example, the platform maintenance personnel can, based on the requirements, set the confidence value that is considered to be "reliably trustworthy" as the first confidence threshold in advance to screen the first classification results that meet the credibility requirements.

[0054] Accordingly, if the execution subject determines that the first classification confidence is greater than or equal to the first confidence threshold, it may choose to continue executing S104.

[0055] S104: Determine the object to be classified as a first data type.

[0056] In an embodiment of the present application, based on the above S103, if the execution subject determines that the first classification confidence is greater than or equal to the first confidence threshold, that is, the first classification confidence indicates that the first classification result for the first modal data is reliable and trustworthy, then the execution subject can determine the object to be classified as the first data type. Thus, the object to be classified is managed and restricted from being directly released.

[0057] In some embodiments, further, if the execution subject determines in S103 that the first classification confidence is less than the first confidence threshold and considers that the classification result for the first modality data may be unreliable, it may choose to continue to execute S105 to further determine the data type of the second modality data.

[0058] S105: Process the second modality data using the data classification model to obtain a second classification result of the second modality data and a second classification confidence level for the second classification result.

[0059] Specifically, the execution subject may process the second modal data using the data classification model in a manner similar to processing the first modal data, so as to obtain a second classification result of the second modal data and a second classification confidence for the second classification result accordingly. The above discussion on the first modal data may be similarly referred to here, and will not be repeated here.

[0060] In some embodiments, if the first modal data is classified as the second data type, then in this case, regardless of whether the result is "reliable and trustworthy", the execution subject may choose to continue to classify and process the second modal data first to determine whether the classification object to be processed will be restricted and managed because the second modal data is classified as the first data type.

[0061] For example, in the process shown in process 100 discussed above, if in the above S102, the executing entity determines that the first classification result indicates that the first modal data is the second data type, then in this case, the executing entity can choose to directly execute the above S105 based on the first modal data being the second data type after completing S102.

[0062] Next, after completing execution of S105 , the execution subject may choose to continue to execute S106 .

[0063] S106, determining whether the second classification result indicates that the second modal data is of the first data type.

[0064] If it is determined that the second classification result indicates that the second modality data is of the first data type, then S107 may be continued to be selected for execution.

[0065] S107: Determine whether the second classification confidence is greater than or equal to a second confidence threshold.

[0066] If the execution determines whether the second classification confidence corresponding to the second classification result is greater than or equal to the second confidence threshold, if it is greater than or equal to, that is, the second modal data can be considered to be "reliable and trustworthy" as the first data type, then the execution subject can continue to choose to execute S108 to similarly determine and classify the object to be classified as the first data type.

[0067] It should be understood that the second confidence threshold can be set with reference to the setting method of the first confidence threshold mentioned above, but the "value" actually used can be the same as or different from the first confidence threshold mentioned above, and the present application is not intended to limit this.

[0068] S108: Determine the object to be classified as a first data type.

[0069] Thus, the execution subject may choose to continue processing and analyzing the second modal data when the first modal data is classified as the first data type and may be unreliable, or when the first modal data is classified as the second data type (or in other words, without considering whether the classification result actually meets the requirements of reliability and trustworthiness). Thus, by processing and analyzing in steps, it is possible to avoid directly and synchronously processing the first modal data and the second modal data, thereby avoiding a large amount of computing resource consumption and wasting computing resources.

[0070] In some embodiments, although the data classification model classifies the first modal data as the first data type, it may also output and "indicate" that the first classification confidence cannot meet the requirements of the first confidence threshold (i.e., the first classification confidence is less than the first confidence threshold, and S105 is executed after S105). In such a case, the first data classification result for the first modal data may not actually be reliably trusted. As discussed above, although the execution subject also chooses to continue to use the data classification model to process the second modal data, it obtains the second classification result of the second modal data and the second classification confidence for the second classification result.

[0071] However, in this process, it may also happen that the second modal data is also classified as the first data type, but the classification result for the second modal data may not be reliably trusted. For example, while the second classification result indicates the first data type, the second classification confidence fails to meet the requirements of the corresponding second confidence threshold. That is, the second classification confidence is actually less than the second confidence threshold. In such a case, the execution subject can determine whether to actually determine and classify the object to be classified as the first data type by comprehensively "evaluating" the first classification confidence and the second classification confidence.

[0072] For this, please refer to Figure 2 , Figure 2 Another process 200 for determining object classification provided by an embodiment of the present application is shown. Process 200 can be continued to be executed by the execution subject when it is determined in S107 shown in the above process 100 that the second classification confidence threshold is less than the second confidence threshold. Specifically, process 200 includes the following steps:

[0073] S201, adding the first classification confidence and the second classification confidence to obtain a summed classification confidence.

[0074] Specifically, the execution subject may choose to add the first classification confidence and the second classification confidence to obtain the summed classification confidence. Generally, the execution subject may choose to directly add the two to obtain the summed classification confidence.

[0075] In some embodiments, considering the possible differences in reference values ​​between different modalities and data volumes, it is also possible to select a weight value corresponding to the modal configuration of the data, and then add the first classification confidence and the second classification confidence by weighted addition to obtain the summed classification confidence.

[0076] As a result, more content may be provided, or analysis weights may be shifted according to analysis needs to meet different analysis needs, using the "summed classification confidence" that meets usage needs and has usage quality.

[0077] S202: In response to the sum of the classification confidence being greater than or equal to a third confidence threshold, determining the object to be classified as a first data type.

[0078] Specifically, for the summed classification confidence, a third confidence threshold can also be configured accordingly, so as to determine whether to determine the object to be classified as the first data type by comparing the summed classification confidence and the third confidence threshold. That is, when the summed classification confidence meets the requirements of the third confidence threshold and is greater than or equal to the third confidence threshold, it is determined that from the "overall dimension", it is reliable and trustworthy to determine the object to be classified as the first data type.

[0079] It should be understood that usually the value of the third confidence threshold will be greater than both the first confidence threshold and the second confidence threshold, thereby "achieving analysis as a whole". For example, in the case of directly adding the original value to the first classification confidence and the second classification confidence, the value of the third confidence threshold can be greater than both the first confidence threshold and the second confidence threshold. However, in scenarios such as the weight addition discussed above, based on the weight setting method, the value of the third confidence threshold may also be less than both the first confidence threshold and the second confidence threshold (for example, the confidence of each classification may be reduced at the same time because the "unit" of the weight setting is too small), or less than one of the first confidence threshold and the second confidence threshold.

[0080] It should be understood that in some embodiments, for the case where both the first modal data and the second modal data may be unreliably classified as the second data type, that is, the first classification result and the second classification result indicate that both the first modal data and the second modal data are classified as the second data type, but the corresponding first classification confidence and the second classification confidence do not meet the corresponding first confidence threshold requirements and second confidence threshold requirements (for example, neither is greater than or equal to the corresponding confidence threshold). The execution subject may also similarly choose to sum the first classification confidence and the second classification confidence, and determine whether the object to be classified can be finally and actually classified as the "second data type" based on whether the sum is greater than the third confidence threshold, or a fourth confidence threshold of a different standard (for example, a higher standard) than the third confidence threshold (for example, the value corresponding to the fourth confidence threshold is greater than the value corresponding to the third confidence threshold).

[0081] In some embodiments, if the execution subject is ultimately unable to reliably classify the object to be classified into the first data type and the second data type (for example, the sum of the classification confidences still cannot meet the requirements of, for example, the third confidence threshold), then in this case, the execution subject can output a prompt message based on this situation and / or provide a classification result of a third data type to provide feedback that the object to be classified cannot be reliably and credibly classified into the first data type or the second data type. Accordingly, for example, the platform can make adaptive decisions based on the feedback information and / or the classification result of the third data type.

[0082] Then, the method for determining the classification of an object provided by the present application uses a data classification model to process the first modal data of the object to be classified, and obtains a first classification result of the first modal data and a first classification confidence for the first classification result, wherein the object to be detected is composed of the first modal data and the second modal data, and the data volume of the first modal data is smaller than that of the second modal data; in response to the first classification result indicating that the first modal data is a first data type, and the first classification confidence is greater than or equal to the first confidence threshold, the object to be classified is determined to be the first data type. In this way, the efficiency of data classification for multimodal data can be improved on the basis of ensuring the data classification quality for multimodal data.

[0083] In some embodiments, although the above-discussed method of configuring data classification models correspondingly for different data modalities can be referred to to implement data processing and classification accordingly. However, such an approach may have problems with high resource configuration requirements and redundant configuration. In this regard, you can choose to configure the data classification model only for one target modality, and use other models and conversion layers to classify the data of each different data modality into the "target modality" for processing. As a result, by separately deploying a data classification model for one data modality, it is possible to provide data classification and processing services for data of different data modalities, saving configuration resources.

[0084] Specifically, the execution subject can configure and train the data classification model based on the target modality (for example, text modality) in advance. Then, the modality encoder can be used to "adjust" the data of each data modality to the data under the target modality. In other words, the execution subject can generate first target modality data corresponding to the first modality data under the target modality in response to the data modality of the first modality data of the object to be classified being not the target modality. Then, in the process of actually using the data classification model to process the first modality data of the object to be classified to obtain the first classification result of the first modality data and the first classification confidence for the first classification result, the execution subject can choose to use the data classification model to process the first target modality data to obtain the first classification result and the first classification confidence for the first modality data.

[0085] Exemplarily, the execution subject can convert the modality using a modality converter configured in association with the object to be classified. The modality converter may include a modality encoder and a modality alignment layer, and the modality encoder can convert image, audio and other data into Embedding vectors. Then, the modality alignment layer in the modality converter can map the Embedding vector from the original modality space to the target modality space such as the text modality, so as to achieve the purpose of "unifying different modality data into the target modality".

[0086] In some embodiments, in the implementation method by converting the data modality, in the process of training the data classification model, the data classification model can also be end-to-end trained in combination with the type unique to the original data modality. For example, the data classification model is trained using the sample input obtained based on the modality conversion and the corresponding sample classification results, so that it can be specifically adjusted for the "input data obtained based on the modality conversion" and has better processing capabilities. For example, for the "features that can be used to determine the classification results" that have a lower probability of appearing in the text and a higher probability of appearing in the image and audio, they are labeled and constructed with unique samples, so that the data classification model has a better recognition ability for the "text content" converted from the features that are unique to the image and audio and can be used to determine the classification results.

[0087] In some embodiments, for the second modal data, the execution subject can also generate second target modal data corresponding to the first modal data under the target modality in response to the data modality of the second modal data of the object to be classified being not the target modality. Then, in S105 shown in the above process 100, the second target modal data is processed using a data classification model as an alternative or alternative to obtain a second classification result and a second classification confidence of the second modal data. In this way, the configuration requirements for the data classification model can be reduced while ensuring the classification quality of multi-modal data, saving computing resources.

[0088] In some embodiments, in order to improve performance, the data classification model can be a large language model. Large Language Model (LLM) is an artificial intelligence model that aims to understand and generate human language. Moreover, the generative language model can perform processing operations accordingly based on the content it understands to obtain corresponding processing results.

[0089] LLMs are characterized by their large size and can often include a large number of parameters to help them learn complex patterns in language data. These models are often based on deep learning architectures such as transformers, which helps them provide better processing performance on various NLP tasks.

[0090] As discussed above, in the embodiments of the present application, the execution subject may also choose to use LLM to improve processing performance. For example, in the above example, in the process of processing modal data of the target modality, LLM may actually be used to process data in the text modality.

[0091] Accordingly, the execution subject can instruct LLM to determine the classification result of the input content and the confidence level of the classification result based on pre-configured guide words, guide labels, etc. For example, in this process, LLM can be based on, for example, "processing the input Embedding vector, analyzing its corresponding data type, outputting the data type and the confidence level of the output result", or, for example, "processing the input Embedding vector, analyzing whether its corresponding data type is the first data type or the second data type, and outputting the confidence level of the judgment result", etc.

[0092] Similarly, for LLM, it can be configured by default to omit the "guide word". For example, for the purpose of outputting the classification result and the corresponding classification confidence, LLM can be based on the default configuration, and of course understand that the input content needs to be classified to determine whether it belongs to the first data type or the second data type, and provide the classification confidence accordingly. Thus, by the default configuration, the generative language model can stably and directionally determine the classification result of the input content and the classification confidence for the classification result.

[0093] Therefore, LLM can be used to process the first modality data and the second modality data of the object to be classified more efficiently and with higher quality, to obtain a first classification result of the first modality data and a first classification confidence for the first classification result, as well as to obtain a second classification result of the second modality data and a second classification confidence for the second classification result.

[0094] In order to deepen the understanding, this application also combines a specific application scenario and exemplarily gives a specific implementation solution, please refer to Figure 3 The architecture 300 of a process that can be used to determine a classification of an object is shown by way of example.

[0095] In the architecture 300 , for example, the object to be classified 310 may be composed of modal data of three data modalities, such as text modality data 313 , image modality data 315 , and audio modality data 317 .

[0096] Accordingly, in the architecture 300, the execution subject can actually be used as the LLM 330 used by the data classification model, and can be configured to process input data with the text modality as the "target modality". Therefore, in the architecture 300, a modality converter 330 can also be configured to process data of other modal types that are not text modalities to obtain data of the corresponding text modality. For example, the modality converter 330 can be used to process data 315 into data 335 of the text modality, and to process data 317 into data 327 of the text modality.

[0097] Therefore, in architecture 300 , the execution entity can use LLM 330 to process the text modal data 313 , 335 and 327 to obtain the classification result 343 corresponding to the data 313 , the classification result 345 corresponding to the data 335 , and the classification result 347 corresponding to the data 327 .

[0098] As discussed above, the LLM 330 may also be configured with a classification head 335 for outputting a classification confidence 353 corresponding to the classification result 343 , a classification confidence 355 corresponding to the classification result 345 , and a classification confidence 357 corresponding to the classification result 347 .

[0099] Architecture 300 can be used to implement the process of determining object classification provided by the embodiments of the present application. For example, when the execution subject determines that the data volume of data 313 is the smallest, the data volume of data 317 is the largest, and the data volume of data 315 is between data 313 and 317, the execution subject first selects the classification result 343 and the corresponding classification confidence 353 obtained based on architecture 300, and then, when it is determined that the classification result 343 is the first data type and the corresponding classification confidence 353 is greater than or equal to the first classification confidence threshold, the object to be classified 310 is classified as the first data type.

[0100] The present application also provides a device for determining object classification, the structure of which is as follows: Figure 4 The device 400 shown. The device 400 includes: a first data processing module 410, configured to process the first modal data of the object to be classified using the data classification model, and obtain a first classification result of the first modal data and a first classification confidence for the first classification result, wherein the object to be detected is composed of the first modal data and the second modal data, and the data volume of the first modal data is smaller than that of the second modal data; a first classification determination module 420, configured to determine the object to be classified as the first data type in response to the first classification result indicating that the first modal data is the first data type, and the first classification confidence is greater than or equal to the first confidence threshold.

[0101] In some embodiments, the device 400 also includes: a second data processing module, configured to, in response to the classification result indicating that the first modal data is the second data type, process the second modal data using a data classification model to obtain a second classification result of the second modal data and a second classification confidence for the second classification result; a second classification determination module, configured to, in response to the second classification result indicating that the second modal data is the first data type and the second classification confidence is greater than or equal to a second confidence threshold, determine the object to be classified as the first data type.

[0102] In some embodiments, the device 400 also includes: a third data processing module, configured to process the second modality data using a data classification model in response to the first classification result indicating that the first modality data is a first data type and the first classification confidence is less than a first confidence threshold, to obtain a second classification result of the second modality data and a second classification confidence for the second classification result.

[0103] In some embodiments, the device 400 also includes: a confidence summation module, configured to add the first classification confidence and the second classification confidence to obtain a summed classification confidence in response to the second classification result indicating that the second modal data is the first data type and the second classification confidence is less than a second confidence threshold; and a third classification determination module, configured to determine the object to be classified as the first data type in response to the summed classification confidence being greater than or equal to the third confidence threshold.

[0104] In some embodiments, the device 400 also includes: a first modality conversion module, configured to generate first target modality data corresponding to the first modality data under the target modality in response to the data modality of the first modality data of the object to be classified is not the target modality; and the first data processing module 410 is further configured to process the first target modality data using a data classification model to obtain a first classification result and a first classification confidence of the first modality data.

[0105] In some embodiments, the device 400 also includes: a second modality conversion module, configured to generate second target modality data corresponding to the first modality data under the target modality in response to the data modality of the second modality data of the object to be classified being not the target modality; and using the data classification model to process the second modality data to obtain a second classification result and a second classification confidence of the second modality data, including: using the data classification model to process the second target modality data to obtain a second classification result of the second modality data and a second classification confidence for the second classification result.

[0106] In some embodiments, the data classification model is a large language model.

[0107] In some embodiments, the first classification confidence is obtained by processing the hidden layer vector output by the last layer decoder in the data classification model by a classification head configured in the data classification model.

[0108] Based on the same inventive concept, an electronic device, a readable storage medium and a computer program product are also provided in an embodiment of the present application. The method corresponding to the electronic device may be the method for determining the classification of an object in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0109] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0110] Figure 5 The structure of an electronic device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the electronic device 500 includes a central processing unit (CPU, Central Processing Unit) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 502 or the program loaded from the storage part 508 to the random access memory (RAM, Random Access Memory) 503. In RAM 503, various programs and data required for system operation are also stored. CPU 501, ROM 502 and RAM 503 are connected to each other through bus 505. Input / output (I / O, Input / Output) interface 504 is also connected to bus 505.

[0111] The following components are connected to the I / O interface 504: an input section 506 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 508 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet.

[0112] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 501, the above functions defined in the method of the present application are executed.

[0113] Another embodiment of the present application further provides a computer-readable storage medium and a computer program product, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application.

[0114] Specifically, the present embodiment can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, systems, devices or devices including but not limited to electricity, magnetism, light, electromagnetic, infrared, or semiconductors, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0115] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0116] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0117] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0120] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules and units is only a logical function division. There may be other division methods in actual implementation. For example, with units as an example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0121] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0122] In addition, each functional module or unit in each embodiment of the present application may be integrated into one processing module or unit, or each module or unit may exist physically separately, or two or more units may be integrated into one module or unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules or units.

[0123] The above-mentioned integrated modules and units implemented in the form of software function modules and units can be stored in a computer-readable storage medium. The above-mentioned software function modules and units are stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform some steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0125] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for determining an object classification, characterized in that include: Processing first modal data of an object to be classified using a data classification model to obtain a first classification result of the first modal data and a first classification confidence for the first classification result, wherein the object to be detected consists of first modal data and second modal data, and a data volume of the first modal data is smaller than that of the second modal data; In response to the first classification result indicating that the first modal data is of a first data type and the first classification confidence is greater than or equal to a first confidence threshold, the object to be classified is determined to be of the first data type.

2. The method according to claim 1, characterized in that Also includes: In response to the classification result indicating that the first modality data is of a second data type, processing the second modality data using the data classification model to obtain a second classification result of the second modality data and a second classification confidence for the second classification result; In response to the second classification result indicating that the second modality data is of the first data type and the second classification confidence is greater than or equal to a second confidence threshold, the object to be classified is determined to be of the first data type.

3. The method according to claim 2, characterized in that Also includes: In response to the first classification result indicating that the first modal data is a first data type and the first classification confidence is less than the first confidence threshold, the second modal data is processed using the data classification model to obtain a second classification result of the second modal data and a second classification confidence for the second classification result.

4. The method according to claim 3, characterized in that Also includes: In response to the second classification result indicating that the second modality data is of the first data type, and the second classification confidence is less than the second confidence threshold, adding the first classification confidence and the second classification confidence to obtain a summed classification confidence; In response to the summed classification confidence being greater than or equal to a third confidence threshold, the object to be classified is determined to be the first data type.

5. The method according to claim 1, characterized in that Also includes: In response to the data modality of the first modality data of the object to be classified not being a target modality, generating first target modality data corresponding to the first modality data in the target modality; as well as The method of processing the first modal data of the object to be classified by using the data classification model to obtain a first classification result of the first modal data and a first classification confidence for the first classification result includes: The first target modality data is processed by using a data classification model to obtain a first classification result and a first classification confidence of the first modality data.

6. The method according to claim 1, characterized in that Also includes: In response to the data modality of the second modality data of the object to be classified not being a target modality, generating second target modality data corresponding to the first modality data in the target modality; as well as Processing the second modality data using the data classification model to obtain a second classification result and a second classification confidence of the second modality data includes: The second target modality data is processed using the data classification model to obtain a second classification result of the second modality data and a second classification confidence for the second classification result.

7. The method according to claim 5 or 6, characterized in that: The data classification model is a large language model.

8. The method according to claim 1, characterized in that The first classification confidence is obtained by processing the hidden layer vector output by the last layer decoder in the data classification model by the classification head configured in the data classification model.

9. A device for determining object classification, characterized in that include: a first data processing module, configured to process first modal data of an object to be classified using a data classification model to obtain a first classification result of the first modal data and a first classification confidence for the first classification result, wherein the object to be detected is composed of first modal data and second modal data, and a data volume of the first modal data is smaller than that of the second modal data; The first classification determination module is configured to determine the object to be classified as the first data type in response to the first classification result indicating that the first modal data is of the first data type and the first classification confidence is greater than or equal to a first confidence threshold.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

11. A computer readable medium, characterized in that Computer program instructions are stored thereon, and the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images