Label determination method, device, equipment, storage medium and product

Abstract

The present application discloses a label determination method, apparatus, device, storage medium and product, which belongs to the field of artificial intelligence technology. The method includes: obtaining a first multimedia object and feature information corresponding to the first multimedia object; performing label recall processing according to the feature information to obtain a label recall result; obtaining a candidate label set based on the second semantic label; and determining the target candidate label in the candidate label set that meets the preset conditions as the first semantic label corresponding to the first multimedia object. In the technical solution provided by the embodiment of the present application, label recall is performed through the feature information of the multimedia object, and high-order semantic labels corresponding to similar multimedia objects can be found and used as candidate labels. Then, high-order semantic labels of multimedia objects can be selected from them according to preset conditions, and high-order semantic labels are determined by semantic text information associated with multimedia information publishing objects, which can accurately reflect semantic information and improve the accuracy of information labels.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a label determination method, apparatus, device, storage medium, and product. Background Art

[0002] In an era of rapid internet development, the threshold for multimedia content creation has been lowered, and the amount of multimedia content uploaded to servers has grown rapidly. Therefore, to reduce the pressure on content services, servers need to send appropriate multimedia content to users based on the content's tags.

[0003] In related technologies, the method of determining labels for multimedia information is mainly to manually browse the multimedia information content to label the multimedia information with corresponding video tags; or to use objective information such as the information category corresponding to the multimedia information, the entity words involved in the multimedia information, etc. as the label of the multimedia information.

[0004] However, the method of manually determining information tags in the above-mentioned related technologies is too costly, and the method of determining information tags using objective information has low accuracy. Summary of the Invention

[0005] The embodiments of the present application provide a label determination method, apparatus, device, storage medium, and product, which can improve the accuracy of information labels and reduce the labor cost of label determination.

[0006] According to one aspect of an embodiment of the present application, a tag determination method is provided, the method comprising:

[0007] Acquire a first multimedia object and feature information corresponding to the first multimedia object;

[0008] performing a tag recall process based on the feature information to obtain a tag recall result; wherein the tag recall result includes a second semantic tag corresponding to at least one second multimedia object in the high-order semantic tag set, the second multimedia object being a multimedia object in at least one multimedia object whose similarity to the first multimedia object is greater than a similarity threshold, and the high-order semantic tag in the high-order semantic tag set is determined based on semantic text information associated with an information publishing object corresponding to the at least one multimedia object;

[0009] Based on the second semantic label, obtaining a candidate label set;

[0010] A target candidate tag that meets a preset condition in the candidate tag set is determined as a first semantic tag corresponding to the first multimedia object, where the first semantic tag is used to represent high-order semantic information of the first multimedia object.

[0011] According to one aspect of an embodiment of the present application, a label determination device is provided, the device comprising:

[0012] an object feature acquisition module, configured to acquire a first multimedia object and feature information corresponding to the first multimedia object;

[0013] a tag recall module, configured to perform tag recall processing based on the feature information to obtain a tag recall result; wherein the tag recall result includes a second semantic tag corresponding to at least one second multimedia object in a high-order semantic tag set, the second multimedia object being a multimedia object among at least one multimedia object whose similarity to the first multimedia object is greater than a similarity threshold, and the high-order semantic tag in the high-order semantic tag set is determined based on semantic text information associated with an information publishing object corresponding to the at least one multimedia object;

[0014] a candidate tag determination module, configured to obtain a candidate tag set based on the second semantic tag;

[0015] The first tag determination module is configured to determine a target candidate tag in the candidate tag set that meets preset conditions as a first semantic tag corresponding to the first multimedia object, where the first semantic tag is used to represent high-order semantic information of the first multimedia object.

[0016] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the above-mentioned label determination method.

[0017] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the above-mentioned label determination method.

[0018] According to one aspect of an embodiment of the present application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to implement the above-described tag determination method.

[0019] The technical solutions provided in the embodiments of the present application can bring the following beneficial effects:

[0020] After obtaining the first multimedia object without a high-order semantic tag and its feature information, label recall can be performed based on the feature information to obtain a label recall result, and the high-order semantic tag corresponding to the second multimedia object similar to the first multimedia object in the label recall result can be used as a candidate tag. The candidate tags can be screened according to preset conditions to determine the high-order semantic tag of the first multimedia object. The labeling process does not require human participation, which reduces labor costs. In addition, the determined high-order semantic tag is determined by the semantic text information associated with the information publishing object corresponding to the multimedia object, which can more accurately reflect the semantic information of the multimedia object, conform to the user's subjective feelings, and improve the accuracy of the information tag. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 is a schematic diagram of an application program operating environment provided by an embodiment of the present application;

[0023] Figure 2 This is the process of the label determination method provided by an embodiment of the present application Figure 1 ;

[0024] Figure 3 A schematic diagram of extracting audio features is exemplarily shown;

[0025] Figure 4 A schematic diagram of extracting text features is shown as an example;

[0026] Figure 5 A schematic diagram of extracting image features is exemplarily shown;

[0027] Figure 6 This is the process of the label determination method provided by an embodiment of the present application Figure 2 ;

[0028] Figure 7 A schematic diagram of a process for determining a content feature vector of a multimedia object is exemplified;

[0029] Figure 8 This is the process of the label determination method provided by an embodiment of the present application Figure 3 ;

[0030] Figure 9 A schematic diagram of a homepage of an information publishing object is shown as an example;

[0031] Figure 10 A schematic diagram showing an exemplary video tag optimization result;

[0032] Figure 11 A schematic diagram of a process for determining a high-level semantic tag set is exemplified;

[0033] Figure 12 A schematic diagram of a process for recalling labels for graphic content objects is shown as an example;

[0034] Figure 13 A schematic diagram of a process for recalling labels for video content objects is exemplified;

[0035] Figure 14 The following is a schematic diagram showing a technical framework of an information flow content service system;

[0036] Figure 15 is a block diagram of a label determination device provided by one embodiment of the present application;

[0037] Figure 16 This is a structural block diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0038] Before introducing the method embodiments provided in the present application, a brief introduction is first given to the relevant terms or nouns that may be involved in the method embodiments of the present application to facilitate understanding by those skilled in the art in the field of the present application.

[0039] Social networks: Social networks originated from online social networking, which began with email. The internet is essentially a network of computers. Early email solved the problem of long-distance email transmission and remains the most popular internet application to this day. It also marked the beginning of online social networking. BBS (Bulletin Board System) took it a step further, normalizing "group messaging" and "forwarding," theoretically enabling the ability to publish information and discuss topics to anyone, becoming a platform for the spontaneous generation of early internet content.

[0040] BBSs advanced online social networking, moving from simply reducing the cost of point-to-point communication to reducing the cost of face-to-face communication. Instant messaging (IM) and blogs are more like upgraded versions of these two social tools. The former improves immediacy (transmission speed) and the ability to communicate simultaneously (parallel processing); the latter begins to embody theories from sociology and psychology: information publishing nodes begin to embody a growing sense of individuality, as scattered information across time can be aggregated and become the "image" and "personality" of the information publishing node. As online social networking quietly evolved, a person's online profile became more complete, and this is when social networks emerged. Social networks encompass all forms of online services centered on human social interaction. The internet is an interactive platform for mutual exchange, communication, and participation. Social networks have expanded the internet from a platform for research, education, and business applications to a tool for human social interaction. Online social networking has even expanded its scope to mobile platforms. Leveraging the ubiquity of mobile phones and the availability of wireless networks, and utilizing various dating, instant messaging, and email software, mobile phones have become the new vehicle for social networking.

[0041] In the era of self-media, diverse voices emerge from all directions, and the voice of "mainstream media" is gradually weakening. People no longer accept being told what's right or wrong by a single "unified voice." Instead, each individual makes their own judgments based on independently acquired information. Self-media differs from information dissemination led by professional media organizations. It is an information dissemination activity led by the general public, transforming traditional "point-to-surface" communication to a "point-to-point" and peer-to-peer communication concept. It also refers to a method of information dissemination that allows individuals to produce, accumulate, share, and disseminate information, combining both privacy and public access. The content structure of self-media is also unique. There's no set core; people write whatever comes to mind and share anything they find valuable. This makes reading excellent self-media articles and graphics truly unique and engaging. They also maintain a strict word count, generally around 1,000 words, allowing users to read smoothly within 10 minutes, making them ideal for quick reading and consumption in the mobile age. They are often displayed in a feed format for users to quickly refresh, and can be considered a new type of information aggregator. Subscriptions are based on friends or public figures they follow, and content is their publicly released updates. When you have a large and active group of friends, you receive constantly updated content, which is the most common form of feed flow. Time is the ultimate dimension of feed flow, as content updates are the result of continuous requests to the server. The timeline is the most primitive, intuitive, and basic display format for feeds. This form of information flow has now become the mainstream way users consume information and news.

[0042] MCN: It is a product form of multi-channel network, which combines PGC (Professional Generated Content) content and, with the strong support of capital, ensures the continuous output of content, thereby ultimately achieving stable commercial realization.

[0043] PGC refers to professionally produced content (such as videos on video sites) and expert-produced content (such as content on social networks). It generally refers to personalized content, diversified perspectives, and virtualized social relationships. It is also known as PPC (Professionally Produced Content).

[0044] Multimedia Object: An individual multimedia object recommended by the information flow service system for users to read and view. It may include graphic objects, image objects, or video objects. Graphic objects are typically user-generated content, including small vertical graphic content and / or short horizontal graphic content, corresponding to the self-media information publishing object. Videos are typically provided by PGC or UGC content producers and ultimately delivered as feeds.

[0045] NetVLAD (NetVector of Locally Aggregated Descriptors) is a scene recognition algorithm that improves on VLAD (Vector of Locally Aggregated Descriptors). The VLAD algorithm uses the scale-invariant feature transform (SIFT) or similar algorithms as its basis to encode the extracted features to obtain a shorter feature string. NetVLAD uses a convolutional neural network as its basic feature extraction structure and connects it to this network to achieve end-to-end training.

[0046] FAISS (Clustering and Similarity Search Library) is an open-source library for clustering and similarity search. It provides efficient similarity search and clustering for dense vectors, supporting searches on billions of vectors. It is currently the most mature approximate nearest neighbor search library. It includes multiple algorithms for searching vector sets of any size, as well as support code for algorithm evaluation and parameter tuning.

[0047] BERT (Bidirectional Encoder Representation from Transformers) is a large-scale text pre-training model. Using a 12-layer transformer encoder, BERT significantly improves baseline performance in natural language processing tasks. Compared to word2vec (word embeddings), BERT, pre-trained on massive amounts of text, can incorporate more transferable knowledge into classification algorithms, providing more accurate text features.

[0048] After briefly introducing the above-mentioned related terms or nouns, the application scenarios involved in this application are introduced below to facilitate understanding by technical personnel in the field of this application.

[0049] In the era of rapid internet development, as the threshold for content production decreases, the volume of multimedia object uploads is growing exponentially. These multimedia objects include various content creation organizations, such as PGC and UGC content from self-media and related creation organizations. For example, information content services based on public accounts on social networks. According to relevant statistics, the peak daily upload volume of multimedia objects written to databases from various sources has exceeded one million in the past period. Optionally, multimedia objects mainly include text content objects, image content objects, graphic and text content objects, audio content objects, video content objects, and audio and video content objects.

[0050] First, let's introduce the distribution process of video content objects (including ordinary videos, short videos, and mini videos). Taking mini videos as an example, the process of video content objects from the beginning of uploading, to successful uploading, and then to successful entry into user consumption is as follows: users shoot videos through terminal shooting tools such as mobile phones, or clients installed on the terminals, such as video clients, social clients, etc., and then upload them through the terminal or client. During the video upload process, the video will be re-transcoded, the video file will be standardized, the metadata of the video will be saved, and the video's playback compatibility on various platforms will be improved. The video will then be manually reviewed, and during the manual review, some auxiliary features of the content will be obtained, such as classification, tags, etc.; then, based on this, manual standardized annotation will be performed to fill in relevant information for the video, such as the video's tags, categories, celebrity information, etc. This is the standardization of content. After activation, a content pool is built and given to the recommendation engine.

[0051] Similarly, for graphic content objects, after the author submits them for publication, after various quality reviews and processing, they will be classified and tag information will be extracted through the NLP (Nature Language processing) algorithm, and then enabled to the content recommendation pool. Finally, the recommendation engine recommends and distributes multimedia content objects in the content pool through recommendation algorithms such as collaborative recommendation, matrix decomposition, supervised learning algorithm Logistic Regression (logistic regression) model, deep learning-based model, Factorization Machine (factor decomposition machine) and GBDT (Gradient Boosting Decision Tree), and users will click on and consume the recommended content. Optionally, the interest information of the target object can be determined based on the interaction data corresponding to the multimedia object, and the label corresponding to the multimedia content object can represent the above interest information.

[0052] The inventors discovered that current labeling is primarily based on manual, objective understanding of the content of multimedia objects, and label quality is also manually determined. Current label quality assessment criteria primarily focus on whether the label objectively reflects the content of the distributed multimedia object. Based on these findings, the inventors discovered the following issues with the labeling of multimedia objects in current information streaming services.

[0053] (1) Some objective labels are not directly helpful for object recommendation, such as labels such as “war” and “movie”.

[0054] (2) There is a lack of advanced tags that can reflect the interest level of the target object and the semantics of the emotional tendency of the content. For example, the interest level information reflected by the label "celebrity" is lower than that of the expression "celebrity marriage", which leads to a poor user experience on the recommendation side. In particular, there is a lack of information tags that can characterize the deep semantics of the content.

[0055] (3) New multimedia objects and new tags emerge in an endless stream, and new hot multimedia objects account for a large proportion of consumption in the information flow, but the corresponding tag extraction and tag coverage are delayed. In the recommendation system, because it needs to meet the needs of different objects, the recommendation system can determine interest demand information based on recognition. The above interest demand information needs to retain complete context information, have complete semantic granularity, and be able to well describe interest tendency information. However, different objects may have different reasons for viewing the same multimedia object, so the tag needs to have a certain degree of generalization ability, but it cannot be too generalized. For example, "star" and "movie" are concepts, but the information they reflect is too broad and cannot accurately describe interest tendency information and deep content semantic information. However, expressions such as "beautiful star", "science fiction movie", "off-road car", and "fuel-efficient car" are more reasonable.

[0056] Therefore, the current standardized manual labeling of simple labels and classification information, that is, traditional classification and labeling, cannot well meet the needs of advanced personalized recommendations. The label accuracy is low, and the cost of manual labeling of such labels is very high.

[0057] In response to the above problems, the present application provides a label determination method that can be applied to an information flow content service system to mark high-order semantic labels for multimedia objects in the information flow content service. The core idea is: by mining and analyzing the semantic text information of the information publishing object corresponding to the multimedia object, such as the publishing object description information and the subject content information of the multimedia object corresponding to the information publishing object, a high-order semantic label dictionary is constructed, so as to update the ordinary labels corresponding to the multimedia object to high-order semantic labels that can represent interest tendency information and emotional semantic information. For multimedia objects that are not assigned high-order semantic labels, high-order semantic labels can be determined for them through recall + sorting, thereby converting the end-to-end classification problem-solving method into an information matching problem. The above recall process may include two recall methods, one is explicit recall, which can recall high-order semantic labels through entities or keywords corresponding to multimedia objects; the other is implicit recall, which can recall high-order semantic labels through feature information of multiple dimensions corresponding to multimedia objects, such as text features, image features, video features, audio features and other feature information corresponding to multimedia objects. After obtaining the recalled multiple tags, the recalled multiple tags are sorted based on their importance in the high-order semantic tag library and their importance in the basic tag library, and the tags with the highest ranking are taken as the final matching result output. Through the tag determination method provided by this application, more high-order semantic tags that can reflect interest tendency information and content semantic information can be mined, and the deep semantic information missing from the basic classification tag information can be supplemented. The system's perspective on the understanding of multimedia object content can be switched from the perspective of entity information understanding to the perspective of high-level abstract information, which comprehensively improves the automation level and efficiency of tag mining. The mined high-order semantic tags can accurately reflect interest tendency information and content semantic information. Ultimately, the distribution efficiency of multimedia objects can be effectively improved based on the above high-order semantic tags.

[0058] The label determination method provided in the embodiment of the present application involves artificial intelligence technology and cloud technology, which is briefly described below to facilitate understanding by those skilled in the art.

[0059] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0060] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0061] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0062] Deep learning: The concept of deep learning originates from the study of artificial neural networks. A multilayer perceptron with multiple hidden layers is an example of a deep learning architecture. Deep learning discovers distributed feature representations of data by combining lower-level features to form more abstract higher-level representations of attribute categories or features.

[0063] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more concretely, it means using cameras and computers instead.

[0064] Machine vision involves the human eye identifying and measuring objects, and then further processing them through graphics to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0065] Key technologies in speech technology include automatic speech recognition (ASR), text-to-speech (TTS), and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech becoming one of the most promising methods of human-computer interaction.

[0066] Natural language processing (NLP) is a key area of research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to linguistics. NLP technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0067] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.

[0068] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand. Cloud computing technology will become a key support. The backend services of technical network systems, such as video websites, image websites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identification mark, which will need to be transmitted to the backend system for logical processing. Different levels of data will be processed separately. All kinds of industry data require strong system support, which can only be achieved through cloud computing.

[0069] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0070] Please refer to Figure 1 , which shows a schematic diagram of an application program running environment provided by an embodiment of the present application. The application program running environment may include: a terminal 10 and a server 20.

[0071] The terminal 10 includes, but is not limited to, electronic devices such as mobile phones, computers, intelligent voice interaction devices, smart home appliances, car terminals, game consoles, e-book readers, multimedia playback devices, wearable devices, etc. The terminal 10 may be installed with a client of an application.

[0072] In the embodiment of the present application, the above-mentioned application can be any application that can provide video information flow content services. Typically, the application is a video application. Of course, in addition to video applications, other types of applications can also provide information flow content services. For example, news applications, social applications, interactive entertainment applications, browser applications, shopping applications, content sharing applications, virtual reality (VR) applications, augmented reality (AR) applications, etc., which are not limited in the embodiment of the present application. In addition, for different applications, the videos they push will also be different, and the corresponding functions will also be different. This can be pre-configured according to actual needs, which is not limited in the embodiment of the present application. Optionally, a client of the above-mentioned application is running in the terminal 10. In some embodiments, the above-mentioned information flow content service covers many vertical content such as variety shows, movies and TV, news, finance, sports, entertainment, games, etc., and users can enjoy a variety of content services such as articles, pictures, small videos, short videos, live broadcasts, special topics, columns, etc. through the above-mentioned information flow content service.

[0073] The server 20 is used to provide background services for the client of the application in the terminal 10. For example, the server 20 can be the background server of the above-mentioned application. The server 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the server 20 provides background services for applications in multiple terminals 10 at the same time.

[0074] Optionally, the terminal 10 and the server 20 may communicate with each other via a network 30. The terminal 10 and the server 20 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0075] Please refer to Figure 2 , which shows the process of the tag determination method provided by an embodiment of the present application Figure 1 This method can be applied to computer equipment, which refers to electronic equipment with data calculation and processing capabilities. For example, the execution subject of each step can be Figure 1 The server 20 in the application running environment is shown. The method may include the following steps (210-240).

[0076] Step 210: Acquire a first multimedia object and feature information corresponding to the first multimedia object.

[0077] Optionally, the first multimedia object is a multimedia object not assigned a high-level semantic tag. The first multimedia object includes but is not limited to a text content object, a picture content object, a graphic content object, an audio content object, a video content object, an audio-video content object, or an audio-video-graphic content object.

[0078] In an exemplary embodiment, the feature information includes at least one search term corresponding to the first multimedia object, and the at least one search term is used to represent the search intent information corresponding to the first multimedia object. In some application scenarios, the application supports a search function, and the terminal can obtain at least one search term based on a preset page. The at least one search term can be a search term entered by the target object in the search box of the preset page. After obtaining the at least one search term, the terminal uses it as input for search or retrieval, and obtains search results corresponding to the at least one search term.

[0079] Optionally, the search results include the first multimedia object, and thus a correspondence between the at least one search term and the first multimedia object may be established.

[0080] Optionally, the first multimedia object is a multimedia object that has been interactively operated in the search results. Similarly, a correspondence between the at least one search term and the first media object can be established. The at least one search term can represent the content semantic information of the first media object, and can also represent the interest tendency information and search intention information of the target object for the first multimedia object. For example, the search term obtained by the terminal is "fuel-efficient car", and the search results obtained after the search are multimedia objects related to "fuel-efficient car". If these multimedia objects related to "fuel-efficient car" have been interactively operated, it can be considered that their content is highly relevant to the search term "fuel-efficient car", and they are multimedia objects that the target object is interested in, and meet the search intention of finding a fuel-efficient car.

[0081] Therefore, the at least one search term corresponding to the first multimedia object may be used as feature information corresponding to the first multimedia object, so as to improve the subsequent recall accuracy and the accuracy of label determination.

[0082] In an exemplary embodiment, the feature information includes entity information and keyword information corresponding to the first multimedia object.

[0083] In a possible implementation, named entity recognition (NER) is performed on the multimedia object to obtain entity information corresponding to the first multimedia object.

[0084] In a possible implementation, information extraction is performed on the multimedia objects to obtain keyword information corresponding to the first multimedia object. Optionally, the information extraction is performed based on a text summary extraction model.

[0085] In an exemplary embodiment, the feature information includes a feature vector of at least one dimension corresponding to the first multimedia object.

[0086] In an exemplary embodiment, a feature vector of at least one dimension corresponding to the first multimedia object may be outputted through a related feature extraction model.

[0087] The feature vector of at least one dimension includes but is not limited to an audio feature vector, a video feature vector, a text feature vector, an entity feature vector, and a cover image feature vector.

[0088] For audio feature vectors, a possible acquisition process is as follows: Videos are divided into many categories, such as tutorials and lectures. The key information in these videos is expressed through audio. Optionally, use VGGish to extract the embedding features of the audio. Figure 3 As shown in FIG, a schematic diagram of extracting audio features is exemplified. First, an audio signal 32 is separated from a video frame sequence 31 corresponding to a multimedia object, and the audio signal 32 is converted into an image input by calculating MFCC (Mel-frequency cepstral coefficients) features. Then, an audio feature sequence 33 is extracted using VGGish. Similar to the video feature sequence, the audio feature sequence can be extracted using NetVLAD 34 corresponding to different shots, and then a global feature vector of the audio modality, i.e., the above-mentioned audio feature vector, is generated through a learnable weight fusion network. VGGish is an audio model trained from the AudioSet dataset, which produces a 128-dimensional embedding feature vector.

[0089] For video feature vectors, one possible acquisition process is as follows: Frame extraction is performed on a video frame sequence to obtain at least one video frame corresponding to the video. Optionally, the frame extraction process is implemented based on TSN (Temporal Segment Networks), and behavior recognition is performed on video frames in the video frame sequence through TSN to obtain the frame extraction result. The at least one video frame is input into a first image feature extraction model to obtain at least one image feature corresponding to the video. Optionally, the first image feature extraction model is Xception. After obtaining the image features corresponding to the at least one video frame, they are input into a second image feature extraction model to obtain a video feature vector. Optionally, the second image feature extraction model is implemented based on the Youtub8M-NeXtVLAD model, and the image feature vectors corresponding to each video frame are output through the intermediate layer of the model. The video feature vector of the video is then obtained by weighted averaging the image feature vectors corresponding to each video frame.

[0090] For text feature vectors, a possible acquisition process is as follows: the text in the video content includes the video title, video summary (the video summary is a rough summary of the video description content) and subtitles. The OCR text in the video can be identified by the OCR (Optical Character Recognition) algorithm, and advertising and marketing content can be filtered out by the advertising and marketing model. After obtaining the text content corresponding to the video, these text information can be encoded. Specifically, the BERT model can be used to generate global feature vectors for the video title, video summary, subtitles and OCR text. The semantic features of the text are extracted by the BERT model, that is, the text string is converted into a vector. Generally, the vector of the penultimate layer of the BERT model is extracted as the text representation vector. If the last layer is extracted, it may deviate from the new task because it is too close to the target.

[0091] In one example, if Figure 4 As shown, it exemplifies a schematic diagram of extracting text features. In some application scenarios, there is a text feature extraction pre-training (Pre-Training) model 410, such as a BERT model. Its pre-training process is to train through sample sentences such as sentence A and sentence B, and the training process is to segment sentences A and sentence B. Get the characters corresponding to sentences A and B. If sentence A has N characters, then its segmentation result is character 1 to character N; if sentence B has M characters, then its segmentation result is character 1 to character M. After word segmentation, add a start character [CLS] in front of the character, and add a separator [SEP] between sentence A and sentence B to obtain a character sequence. Then extract the features corresponding to each character, and the character feature corresponding to the above-mentioned start character [CLS] is E[CLS] ; The character features corresponding to characters 1 to N of sentence A are E1 to E N ; The character feature corresponding to the separator [SEP] is E [SEP] ; The character features corresponding to characters 1 to M of sentence B are E1 1 to E M 1 Next, based on the characteristics of each character, the latent features of each character are extracted. The character latent feature corresponding to the above-mentioned start character [CLS] is C; the character latent features corresponding to characters 1 to N of sentence A are T1 to T N ; The character hidden feature corresponding to the separator [SEP] is T [SEP] ; The character hidden features corresponding to characters 1 to M of sentence B are T1 1 To T M 1 Finally, text feature vectors for sample sentences such as Sentence A and Sentence B are generated based on the aforementioned latent features. The pre-trained text feature extraction model 410 already has a certain level of model accuracy and can be fine-tuned to incorporate more transfer knowledge to obtain a more accurate text feature extraction model 420 that can be applied to different text feature extraction tasks, outputting more precise text feature expressions.

[0092] Among them, the text feature extraction model 420 is a fine-tuned text feature extraction model adopted in the embodiment of the present application. The text content corresponding to the multimedia object, such as sentence C and sentence D, can be input into the text feature extraction model 420. Based on the same method mentioned above, the text feature vector corresponding to the multimedia object can be obtained.

[0093] For the cover image feature vector (or image feature vector), a possible acquisition process is as follows: it can be extracted through the image feature extraction model. In an example, Figure 5 As shown in FIG, it exemplarily shows a schematic diagram of extracting image features. Figure 5 In the illustrated process, a multimedia object's cover image 51 can be input into an image feature extraction model to obtain a cover image feature vector 52. Such image feature extraction models include, but are not limited to, classic image classification models such as VGG16, the Inception series of models, and ResNet. Optionally, Inception-ResNet v2 is selected as the image feature extraction model.

[0094] For entity feature vectors, a possible acquisition process is as follows: Named Entity Recognition (NER) processing can be performed on the content of the multimedia object to obtain the entity word corresponding to the first multimedia object. The entity word feature vector corresponding to the entity word after passing it through BERT can be used as the entity word feature vector.

[0095] Step 220: perform label recall processing based on the feature information to obtain a label recall result.

[0096] Among them, the tag recall result includes a second semantic tag corresponding to at least one second multimedia object in the high-order semantic tag set, the second multimedia object is a multimedia object in at least one multimedia object whose similarity with the first multimedia object is higher than a similarity threshold, and the high-order semantic tag in the high-order semantic tag set is determined based on the semantic text information associated with the information release object corresponding to the at least one multimedia object.

[0097] There is a corresponding relationship between the multimedia object and the information publishing object. The above-mentioned information publishing object can be the original publishing account, forwarding account, etc. corresponding to the multimedia object.

[0098] The semantic text information corresponding to the aforementioned information publishing object is used to represent the content semantic information and content interest trend information of the multimedia object corresponding to the information publishing object, including but not limited to the publishing object description information corresponding to the information publishing object, the content theme information corresponding to the target multimedia object corresponding to the information publishing object, and search information. The aforementioned semantic text information, such as the publishing object description information, content theme information, and search information, often contains more semantic information, or in other words, the connotation information of the content. Therefore, high-level semantic tags can be mined with the help of the aforementioned semantic text information. The aforementioned semantic text information can be obtained from a large amount of corpus information on the Internet and can be manually reviewed before use.

[0099] The high-order semantic tags in the above-mentioned high-order semantic tag set are determined based on the semantic text information associated with the information publishing object corresponding to at least one multimedia object. Therefore, the above-mentioned high-level tags can better represent the fine-grained semantic feature information that the content of the multimedia object itself wants to express, and are easier to associate with interest tendency information, thereby improving the distribution efficiency of multimedia objects.

[0100] Compared with basic tags that can only depict the objective information of multimedia objects but cannot express the semantic information of the content, high-order semantic tags determined by the semantic text information associated with the above-mentioned information publishing objects can switch the system's understanding of the content of multimedia objects from the perspective of entity information understanding to the perspective of high-level abstract information, so as to achieve label optimization of multimedia objects and transition from basic tags to high-order semantic tags that can represent interest, emotional tendencies and content semantic information.

[0101] In an exemplary embodiment, to generate the above-mentioned high-level semantic tag set, such as Figure 6 As shown, the above method further includes the following steps (250-280): Figure 6 The process of the tag determination method provided by an embodiment of the present application is shown Figure 2 .

[0102] Step 250: clustering the at least one multimedia object to obtain at least one multimedia object set and at least one information publishing object set corresponding to the at least one multimedia object set.

[0103] In an exemplary embodiment, at least one multimedia object and content feature information corresponding to the at least one multimedia object are obtained. Optionally, the at least one multimedia object is an existing multimedia object. The content feature information includes a content feature vector corresponding to the multimedia object.

[0104] In one example, if Figure 7 As shown in FIG, it exemplarily shows a flow chart of determining a feature vector of multimedia object content. Figure 7 In the example, the multimedia object is a video, and the content feature vector of the video is a video content embedding vector. The video content embedding vector refers to a low-dimensional vector that represents the content features of the video. The "distance" between two video content embedding vectors represents the distance between the two videos, and the content similarity between the videos can be calculated. The video content embedding vector here can be understood as an "implicit" feature based on the content. The video content embedding vector contains two layers of meaning. The first layer of meaning: the low-dimensional dense features that can represent the video content obtained through representation learning, which can be specifically expressed as a one-dimensional array. For example, the video content embedding vector is composed of 128 float (floating point) data; the second layer of meaning: the low-dimensional dense features that can represent the video content obtained through metric learning, so the video content embedding vector is a vector that is convenient for similarity measurement. The "distance" between two video content embedding vectors represents the "content similarity" between the two videos. As shown in the figure, Figure 7As shown, a video frame sequence 71 is subjected to frame extraction processing to obtain at least one video frame 72 corresponding to the video. Optionally, the frame extraction processing is implemented based on TSN (Temporal Segment Networks), and behavior recognition is performed on the video frames in the video frame sequence through TSN to obtain the frame extraction result. The at least one video frame 72 is input into a first image feature extraction model to obtain at least one image feature 74 corresponding to the video 72. Optionally, the first image feature extraction model is Xception. After obtaining the image feature 74 corresponding to the at least one video frame 72, it is input into a second image feature extraction model to obtain a video content embedding vector 75. Optionally, the second image feature extraction model is implemented based on the Youtub8M-NeXtVLAD model, and the image feature vector corresponding to each video frame 72 is output through the model's intermediate layer. The image feature vector corresponding to each video frame 72 is then weighted averaged to obtain the video content embedding vector 75 of the video.

[0105] Based on the content feature information, clustering processing is performed on at least one multimedia object to obtain at least one multimedia object set.

[0106] In one possible implementation, at least one multimedia object may be clustered on a content dimension by determining vector similarities between content feature vectors corresponding to each multimedia object, thereby obtaining at least one multimedia object set. The multimedia objects within each multimedia object set are similar to each other. The vector similarities between the content feature vectors of similar multimedia objects are less than or equal to a vector similarity threshold.

[0107] In one possible scenario, the content feature information may be an object grouping identifier corresponding to at least one multimedia object. The at least one multimedia object may be clustered based on the object grouping identifier to obtain at least one multimedia object set. The multimedia objects within each multimedia object set are similar to each other. The object grouping identifiers between similar multimedia objects may be the same or similar.

[0108] At least one information publishing object set corresponding to at least one multimedia object set is determined.

[0109] In some practical application scenarios, multimedia objects and information publishing objects have a corresponding relationship. Based on the corresponding relationship between multimedia objects and information publishing objects, the information publishing object corresponding to each multimedia object in at least one multimedia object set can be determined, thereby obtaining at least one information publishing object set corresponding to the at least one multimedia object set.

[0110] Step 260: Obtain semantic text information associated with at least one information release object in the information release object set.

[0111] In one possible implementation, semantic text information associated with each information release object in at least one information release object set is searched. Optionally, the semantic text information associated with the information release object is obtained from a metadata database. Optionally, the information display page corresponding to the information release object is intercepted, and the semantic text information associated with the information release object is obtained from the information display page, such as by identifying the semantic text information associated with the information release object through optical character recognition (OCR). Optionally, the information stream corresponding to the information release object is obtained, and the semantic text information associated with the information release object is obtained from the information stream.

[0112] In an exemplary embodiment, as Figure 8 As shown, the implementation process of the above step 260 may include the following steps (261-264): Figure 8 The process of the tag determination method provided by an embodiment of the present application is shown Figure 3 .

[0113] Step 261: When the semantic text information includes publishing object description information, obtain the publishing object description information corresponding to the information publishing object in at least one information publishing object set.

[0114] The above-mentioned publishing object description information is used to represent the attribute information of the above-mentioned information publishing object, such as the introduction information and subject information of the above-mentioned information publishing object. The above-mentioned publishing object description information can represent the content semantic information and interest tendency information of the multimedia object corresponding to the information publishing object.

[0115] Step 262: When the semantic text information includes content theme information, determine the target multimedia object corresponding to the information publishing object.

[0116] The aforementioned content theme information refers to the content body information corresponding to the target multimedia object corresponding to the information publishing object. Optionally, the aforementioned target multimedia object is the multimedia object published by the information publishing object. In some practical scenarios, an information publishing object may correspond to multiple multimedia objects that it publishes. Some information publishing objects may only correspond to multiple multimedia objects of a certain content theme. Of course, the multimedia objects corresponding to an information publishing object can also correspond to multiple content themes. The aforementioned content theme information can represent the content semantic information and interest tendency information of the multimedia object.

[0117] Therefore, first, it is necessary to determine the target multimedia object corresponding to the information publishing object based on the correspondence between the information publishing object and the multimedia object, and then execute step 263.

[0118] Step 263: Obtain content theme information corresponding to the target multimedia object.

[0119] After determining the target multimedia object corresponding to the information publishing object, content theme information corresponding to the target multimedia object may be determined, such as group name information and group theme information corresponding to the target multimedia object.

[0120] In one example, if Figure 9 As shown, it exemplarily shows a page diagram of the homepage of an information publishing object. On page 91, the avatar picture, nickname information, profile information of the information publishing object 1 and multiple multimedia objects 912 corresponding to the information publishing object 1 are displayed. Optionally, the above-mentioned multimedia object 912 is a video, and the content theme information corresponding to the above-mentioned multiple multimedia objects 912 is the same content theme, and the corresponding content theme information is "Couple Series". On page 92, the avatar picture, nickname information, profile information and multiple multimedia objects 922 corresponding to the information publishing object 2 are displayed. Optionally, the above-mentioned multimedia object 922 is a video, and the content theme information page corresponding to the above-mentioned multiple multimedia objects 922 is the same content theme, and the corresponding content theme information is "Father Series". Page 93 is the display page corresponding to the multimedia object collection, and the content theme information corresponding to the multimedia object collection is "Mother's Love Series".

[0121] Step 264: When the semantic text information includes search information, obtain the search information associated with the target multimedia object.

[0122] The search results corresponding to the search information may include the target multimedia object. Therefore, after determining the target multimedia object, the search information corresponding to the object identifier may be obtained based on the object identifier corresponding to the target multimedia object. The search information may include search information of at least one modality corresponding to the target multimedia object, including but not limited to search terms, search sentences, search images, search audio, and search videos.

[0123] Step 270: Obtain at least one high-level semantic tag corresponding to at least one multimedia object set based on the semantic text information.

[0124] In an exemplary embodiment, based on the semantic textual information associated with the information release objects in the at least one information release object set, at least one high-order semantic tag corresponding to the at least one information release object set can be determined; based on the correspondence between the at least one information release object set and the at least one multimedia object set, the at least one high-order semantic tag corresponding to the at least one information release object set can be determined as the at least one high-order semantic tag corresponding to the at least one multimedia object set. Optionally, the high-order semantic tag corresponding to each multimedia object set can be assigned to the multimedia objects within that multimedia object set.

[0125] In one possible implementation, the semantic text information can be sent to a manual review system, where the high-level semantic tags can be manually summarized based on the semantic text information. After the high-level semantic tags are manually summarized, the summarized high-level semantic tags can be received. By determining the semantic text information associated with information release objects in a set of multiple information release objects, the efficiency, accuracy, and label coverage of manually summarizing high-level semantic tags can be improved.

[0126] In another possible implementation, the above-mentioned semantic text information can be input into a target machine learning model, and the target machine learning model can be used to perform information extraction on the above-mentioned semantic text information to obtain high-order semantic labels.

[0127] In one example, if Figure 10 As shown, a schematic diagram of the video tag optimization results is shown. Before tag optimization, the common tags corresponding to video 101 included: "Program 1 Name, Star 1 Name, Star 2 Name, Program 1 Genre, Comedy, Mainland Variety Show, Variety Show Clip." After tag optimization, the high-level semantic tags added to video 101 included: "Star 1 Marriage." Before tag optimization, the common tags corresponding to video 102 included: "Comedy Drama, Family Drama, Reversal Drama, Couple Drama." After tag optimization, the high-level semantic tags added to video 102 included: "Tease the Boyfriend Series." Before tag optimization, the common tags corresponding to video 103 included: "Reptiles, Crocodiles, Cub, Lemur, Animal Science." After tag optimization, the high-level semantic tags added to video 103 included: "Being a Mom is Not Easy." Before tag optimization, the common tags corresponding to video 104 included: "Eating Broadcast, Foodie, Autonomous Sensory Meridian Response (ASMR), Ice Cream." After tag optimization, the high-level semantic tags added to video 104 included: "What Should It Be."

[0128] Step 280: Obtain a high-order semantic tag set based on at least one high-order semantic tag.

[0129] In one example, if Figure 11 As shown, it exemplifies a flow chart of determining a high-level semantic tag set. By obtaining existing multimedia objects corresponding to multiple information release objects (information release object 1 to information release object x), the above-mentioned existing multimedia includes works 1-1, works 1-2, ..., works 1-n corresponding to information release object 1; ...; works x-1, works x-2, ..., works xm corresponding to information release object x. By clustering the above-mentioned existing multimedia objects, multiple work sets can be obtained. Based on the information release objects corresponding to the works in the multiple work clusters, multiple information release object sets corresponding to the multiple work sets can be obtained, thereby obtaining the release object description information and content theme information of the information release objects in the multiple information release object sets. Through the above-mentioned release object description information and content theme information, high-order semantic tags can be extracted to obtain a high-order semantic tag set, that is, a high-order semantic tag dictionary.

[0130] In one possible implementation, the method for extracting high-level semantic tags can be manual summarization. By determining the publishing object description information and content theme information of the information publishing objects in the set of multiple information publishing objects, the efficiency of manually summarizing the high-level semantic tag dictionary and the accuracy and coverage of the tags can be improved.

[0131] In another possible implementation, the above-mentioned method of extracting high-order semantic tags can also be to extract information from the publishing object description information and content subject information of the information publishing objects in the above-mentioned multiple information publishing object sets through a machine learning model to obtain high-order semantic tags.

[0132] In the subsequent process, the newly released multimedia objects in the information flow content library can be associated with the high-order semantic tag dictionary through approximate recall and sorting, so as to label the newly entered multimedia objects with appropriate high-order semantic tags.

[0133] In one embodiment above, the feature information includes entity information and keyword information corresponding to the first multimedia object. Figure 6 As shown, the implementation process of the above step 220 may include the following steps (221-222).

[0134] Step 221 : performing a comparison and recall process on the entity information and keyword information corresponding to the first multimedia object and the entity information and keyword information corresponding to at least one multimedia object to obtain a first recalled object.

[0135] The at least one second multimedia object includes a first recall object.

[0136] In one possible implementation, the entity information corresponding to the first multimedia object is compared with the entity information corresponding to at least one multimedia object to obtain a multimedia object similar to the first multimedia object in terms of entity information. The keyword information corresponding to the first multimedia object is compared with the keyword information corresponding to at least one multimedia object to obtain a multimedia object similar to the first multimedia object in terms of keyword information. The multimedia objects similar to the first multimedia object in terms of both entity information and keyword information are screened to obtain a first recall object. The first recall object may be a multimedia object in the at least one multimedia object that is similar to the first multimedia object in terms of both entity information and keyword information.

[0137] Step 222: Obtain a label recall result based on the high-order semantic label corresponding to the first recalled object in the high-order semantic label set.

[0138] In one embodiment above, the feature information includes a feature vector of at least one dimension corresponding to the first multimedia object. Figure 6 As shown, the implementation process of the above step 220 may further include the following steps (223-224).

[0139] Step 223 : performing a comparison and recall process on the feature vector of at least one dimension corresponding to the first multimedia object and the feature vector of at least one dimension corresponding to at least one multimedia object to obtain a second recalled object.

[0140] The at least one second multimedia object includes a second recall object.

[0141] Semantic similarity can be measured by the distance between feature vectors. Semantic similarity has many important application scenarios, such as being used for semantic recall in retrieval systems or as a feature for precise ranking.

[0142] The recall process described above leverages the FAISS framework to build a distributed, high-dimensional nearest neighbor retrieval platform. This platform can recall feature vectors of various dimensions from tens of millions of vectors in tens of milliseconds. These include the first m (an integer greater than 0) similar multimedia objects in text, audio, video, and cover image feature vectors, or similar multimedia objects with similarity scores above a cutoff threshold. Multi-dimensional recall can effectively improve the overall recall rate. In some application scenarios, similarity retrieval of text feature vectors in the text feature dimension based on a text semantic similarity model can assist in text classification and mitigate the long iteration cycles associated with classification model updates.

[0143] In an exemplary embodiment, the feature vector of at least one dimension corresponding to the first multimedia object is compared with the feature vector of at least one multimedia object in the corresponding dimension to obtain feature vector similarity.

[0144] Obtain multimedia objects whose feature vector similarity is higher than a feature vector similarity threshold in at least one multimedia object, obtain similar multimedia objects corresponding to the first multimedia object in at least one dimension, and filter out multimedia objects that are similar to the first multimedia object in a target number of dimensions from the similar multimedia objects, and use them as the second recall objects.

[0145] Step 224 : Obtain a label recall result based on the high-order semantic label corresponding to the second recalled object in the high-order semantic label set.

[0146] The following is an example to illustrate the above recall process.

[0147] In one example, if Figure 12 As shown, it exemplarily shows a flow chart of recalling labels for graphic content objects. Figure 12 The multimedia object 121 shown is a graphic content object. By performing named entity recognition processing on the title content and body content of the multimedia object 121, the entity word corresponding to the multimedia object 121 can be obtained, such as "car name". By performing information extraction processing on the title content and body content of the multimedia object 121, keyword information can be obtained, such as "fuel consumption 5 liters". The search term corresponding to the above multimedia object 121 is "fuel-efficient family car". The correspondence between "car name" and "fuel-efficient family car" can be constructed based on relational recall, and the correspondence between "fuel consumption 5 liters" and "fuel-efficient family car" can be constructed based on semantic recall. Optionally, the above "fuel-efficient family car" can be used as a high-level semantic label corresponding to the multimedia object 121.

[0148] In one example, if Figure 13 As shown, it exemplarily shows a flow chart of recalling labels for video content objects. Figure 13 The multimedia object 131 shown is a video, and the feature information corresponding to the video includes features in at least one dimension. The features in the at least one dimension corresponding to the multimedia object 131 include, but are not limited to, video features, audio features, fusion features, text features, entity features, and cover image features.

[0149] The above-mentioned video features can be the video content feature vectors mentioned above; the audio features are usually VGGish audio features; the text features can be text feature vectors that use Bow to represent the title or text content of the multimedia object, which can be used for recall; for entity features, the entity word feature vector corresponding to the entity word after passing through BERT can be directly used, and can be used for recall; the cover image feature can be the image feature vector corresponding to the cover, and similar cover images can be determined through the image feature vector during recall, and the recalled label can also be the high-order semantic label corresponding to the similar cover image.

[0150] exist Figure 13 In the example, a correspondence-based recall can be performed based on the entity features corresponding to multimedia object 131, and concepts or topics corresponding to similar entity features can be output as high-order semantic tags. Semantic recall can also be performed based on the video features, audio features, fusion features, text features, and cover image features corresponding to multimedia object 131, and concepts and topics corresponding to similar semantic features can be input as high-order semantic tags corresponding to multimedia object 131. The above concepts and topics can be high-order semantic tags in a high-order semantic tag library.

[0151] Step 230: Obtain a candidate tag set based on the second semantic tag.

[0152] In one embodiment above, the feature information includes at least one search term corresponding to the first multimedia object. Figure 6 As shown, the above step 230 can be replaced by the following step 231.

[0153] Step 231: Obtain a candidate tag set based on the second semantic tag and at least one search term.

[0154] Optionally, the above-mentioned at least one search term can be used as a label to mark the first multimedia object from the user's perspective, which can reflect the content semantic information of the first multimedia object and the interest tendency information and search intention information of the target object. Therefore, the above-mentioned at least one search term can be used as a candidate label for screening and marking in subsequent processes.

[0155] Step 240 : Determine the target candidate tag that meets the preset conditions in the candidate tag set as the first semantic tag corresponding to the first multimedia object.

[0156] The first semantic tag is used to represent high-level semantic information of the first multimedia object.

[0157] In a possible embodiment, after obtaining the above-mentioned tag recall result and generating the above-mentioned candidate tag set based on the second semantic tag corresponding to the second multimedia object in the tag recall result, the target candidate tag can be selected based on the importance of the candidate tag. Optionally, the overall importance data of the candidate tag is determined, and the above-mentioned overall importance data can be positively correlated with the importance of the candidate tag. Accordingly, the above-mentioned preset conditions include that the overall importance data is higher than or equal to the preset importance threshold. Therefore, the candidate tag in the candidate tag set whose overall importance data is higher than or equal to the preset importance threshold can be determined as the target candidate tag, and it can be determined as the first semantic tag corresponding to the first multimedia object.

[0158] In another possible embodiment, at least one search term in the candidate tag set is determined as the first semantic tag corresponding to the first multimedia object. Accordingly, the preset condition includes the tag term being a search term, so at least one search term in the candidate tag set can be determined as the target candidate tag and the first semantic tag corresponding to the first multimedia object.

[0159] In an exemplary embodiment, as Figure 6 As shown, the implementation process of the above step 240 may include the following steps (241-245).

[0160] Step 241 : determining first importance data corresponding to candidate tags in the candidate tag set in the high-order semantic tag set, and second importance data corresponding to the candidate tags in the target tag set.

[0161] In an exemplary embodiment, the overall importance of the candidate tags can be determined based on the importance of the candidate tags in the high-order semantic tag set and the importance of the candidate tags in the target tag set. The first importance data is used to characterize the importance of the candidate tags in the high-order semantic tag set, and the second importance data is used to characterize the importance of the candidate tags in the target tag set. The target tag set can be a global tag set, including all basic tags and high-order semantic tags.

[0162] In one possible implementation, the first importance data includes a first probability value corresponding to a word in the candidate tag in a set of high-order semantic tags, and the second importance data includes a second probability value corresponding to a word in the candidate tag in a set of target tags.

[0163] Optionally, the first probability value corresponding to the word in the candidate tag in the high-order semantic tag set is determined, and the above-mentioned first importance data is obtained based on the first probability value corresponding to the word in the candidate tag in the high-order semantic tag set.

[0164] Optionally, a second probability value corresponding to the word in the candidate tag in the target tag set is determined, and the above-mentioned second importance data is obtained based on the second probability value corresponding to the word in the candidate tag in the target tag set.

[0165] Step 242: Determine the overall importance data corresponding to the candidate tag based on the first importance data and the second importance data.

[0166] The overall importance data is used to represent the overall importance of candidate tags.

[0167] The importance of the tag word corresponding to the candidate tag increases proportionally with the number of times it appears in the target set, but decreases inversely with the frequency of its appearance in the corpus. Therefore, the overall importance data corresponding to the candidate tag can be determined based on the above first importance data and second importance data.

[0168] In one possible implementation, Figure 8 As shown, the implementation process of the above step 242 may include the following steps (2421-2422).

[0169] Step 2421: Determine the importance difference data between the first importance data and the second importance data.

[0170] In one possible implementation, for each word in a candidate tag, after calculating each word's first probability value in the higher-order semantic tag set and each word's second probability value in the target tag set, a probability difference between the first and second probability values corresponding to each word can be calculated. This probability difference can also reflect the importance of the word between the higher-order semantic tag set and the target tag set.

[0171] Correspondingly, the above importance difference data includes the probability difference corresponding to each word.

[0172] If the target probability difference corresponding to the target word is greater than 0, it means that the target word is more important in the high-order semantic tag set. If the target probability difference corresponding to the target word is less than or equal to 0, it means that the target word is more important in the target tag set.

[0173] Compare the target probability difference corresponding to the current word with 0. If the target probability difference corresponding to the current word is greater than 0, the target probability difference will be accumulated, and the next word corresponding to the current word will be updated to the current word, and the step of comparing the target probability difference corresponding to the current word with 0 will be restarted; if the target probability difference corresponding to the target word is less than or equal to 0, the target probability difference will not be accumulated, and the next word corresponding to the current word will be updated to the current word, and the step of comparing the target probability difference corresponding to the current word with 0 will be restarted; until the current word is the last word. After the loop ends, the target importance difference corresponding to the target word in the candidate tag can be obtained, that is, the probability difference corresponding to the target word and the target word. The above-mentioned target word is a word with a first probability value greater than a second probability value. The above-mentioned target importance difference represents the difference in importance of the target word in the high-order semantic tag set that is higher than the importance of the target word in the target tag set. The above-mentioned importance difference data includes the above-mentioned target importance difference.

[0174] Step 2422: Determine the overall importance data based on the importance difference data and the first importance data.

[0175] The first probability values corresponding to the words in the candidate tags are summed up to obtain the accumulated value of the first probability values corresponding to the candidate tags.

[0176] The ratio between the target importance difference and the accumulated value of the first probability value is determined, and based on the ratio, the importance index data corresponding to the candidate tag can be obtained.

[0177] In a possible implementation, the calculation formula for the above importance index data is as follows:

[0178]

[0179] Here, sim(c, d) is the importance indicator data, which is negatively correlated with the overall importance of the candidate tags. c refers to the set of high-order semantic tags, t refers to each word in the candidate tags, d refers to the target tag set, μ(t|c) refers to the first probability value corresponding to word t in the high-order semantic tag set c, and μ(t|d) refers to the second probability value corresponding to word t in the target tag set d.

[0180] According to the negative correlation between the importance index data and the overall importance of the candidate tags, the overall importance of the candidate tags can be determined. The overall importance data includes the overall importance of the candidate tags.

[0181] Step 243: Arrange the candidate tags in descending order according to the overall importance data to obtain an arrangement result.

[0182] Arrange the candidate tags in descending order according to their overall importance to obtain the arrangement result.

[0183] Step 244: determine the first n candidate tags in the arrangement result as target candidate tags.

[0184] n is an integer greater than 0.

[0185] Correspondingly, the above-mentioned preset conditions include the top n candidate tags in the sorting result.

[0186] Step 245: Determine the target candidate tag as the first semantic tag.

[0187] To sum up, the technical solution provided by the embodiment of the present application, after obtaining the first multimedia object without high-order semantic tags and its feature information, can perform label recall based on the feature information to obtain the label recall result, and use the high-order semantic tag corresponding to the second multimedia object similar to the first multimedia object in the label recall result as the candidate tag, and screen the candidate tags through preset conditions to determine the high-order semantic tag of the first multimedia object. The label marking process does not require human participation, which reduces labor costs, and the determined high-order semantic tag is determined by the semantic text information associated with the information release object corresponding to the multimedia object, which can more accurately reflect the semantic information of the multimedia object, conform to the user's subjective feelings, and improve the accuracy of the information tag.

[0188] In an exemplary embodiment, the above tag determination method can be applied to an information flow content service system. Figure 14 As shown, it exemplarily shows a technical framework diagram of an information flow content service system. Figure 14 The various service modules and their main functions in the information flow content service system are shown as follows.

[0189] 1. Content production and consumption

[0190] (1) PGC or UGC and MCN content producers provide publishing portals for multimedia objects, such as video content objects, graphic content objects, and atlas content objects, through mobile terminals or back-end API (Application Programming Interface) systems. The multimedia objects obtained by the above publishing portals are the main content sources of information flow content services.

[0191] (2) The content production end uploads and publishes multimedia objects through communication with the upstream and downstream content interface services. The content production end can first obtain the interface address of the upload server and then upload the local file. Among them, video content publishing is usually a terminal with a shooting function as a camera end. During the shooting process, the user can select matching music for the local video content, perform corresponding editing, select cover images, filter templates and video beautification functions, etc. Graphic content is usually published through a graphic editor and typesetting system.

[0192] (3) The content consumer obtains the index information of the multimedia object by communicating with the upstream and downstream content interface services, and directly obtains the multimedia object source file from the content storage service based on the above index information, and then loads the multimedia object source file for display to the user. The above index information can be the index information of the multimedia objects subscribed to by the user. The content storage server stores the multimedia object entities, such as video source files and cover image source files, while the metadata of the multimedia objects, such as title, author, cover image, category, tag information, etc., is stored in the content database.

[0193] (4) The content production end and the content consumption end simultaneously report the log data, freezes, loading time, play clicks and other data during the upload and download process to the upstream and downstream content interface servers or other background servers for subsequent data statistical analysis.

[0194] (5) The content consumption end usually displays content to users through the feed stream so that users can browse and consume content data.

[0195] 2. Uplink and Downlink Content Interface Server

[0196] (1) Communicate directly with the content production end to obtain the data submitted by the content production end, which usually includes metadata such as the title of the multimedia object, the information release object, the summary, the cover image, and the release time.

[0197] (2) Writing metadata into the content database, such as file size, cover image link, title, release time, information release target, etc.

[0198] (3) Synchronize the multimedia objects submitted by the content production end to the dispatch center server (in Figure 14 The object enters the dispatch center) so that the dispatch center server can perform subsequent multimedia object processing and flow, including quality processing and content understanding processing, such as mining high-order semantic tags of multimedia objects.

[0199] 3. Content Database

[0200] (1) The content database is the core database of multimedia objects. The metadata of all multimedia objects published by all content production ends is stored in this content database, focusing on storing the metadata of the multimedia objects themselves, such as file size, cover image link, bit rate, file format, title, release time, information release object, file size, file format, original mark, first release mark, and classification label information of multimedia objects during the manual review process. The above classification label information includes first-level, second-level, and third-level classification and label information. For example, in a video explaining a certain brand of mobile phone, the first-level classification is technology, the second-level classification is smart phones, and the third-level classification is domestic mobile phones. The label information can include basic labels, such as a certain brand and model, and can also include high-level semantic labels, such as "power-saving mobile phone", "long standby time", "the most popular mobile phone for a certain group of people", etc.

[0201] (2) The upstream and downstream content interface services will perform standard transcoding operations when receiving multimedia object metafiles. After transcoding is completed, the meta information will be asynchronously returned, mainly including file size, bit rate, specifications, and captured cover images. These meta information will be stored in the content database.

[0202] (3) The manual review system will read the information in the content database during the manual review process, and the manual review results and status will also be returned to the content database by the manual review system.

[0203] (4) The content processing by the dispatch center server mainly includes machine processing and manual review processing. The core process of machine processing here includes calling the content deduplication service to identify completely repeated and similar content. The deduplication results will be written into the content database. Completely repeated content will not be manually processed again, saving human resources for review.

[0204] 4. Dispatch Center Server

[0205] (1) The scheduling center server is responsible for the entire scheduling process of multimedia object flow, receives multimedia objects through the uplink and downlink content interface services, and then obtains the metadata of the multimedia objects from the content database.

[0206] (2) Schedule manual review system and precision verification service to control the order and priority of scheduling.

[0207] (3) For multimedia objects, first communicate with the recall retrieval service, and then communicate with the content deduplication service to identify duplicate or similar multimedia objects.

[0208] (4) If there are no repeated or similar multimedia objects, the content similarity and similarity relationship chains between the multimedia objects are output for the recommendation system to use.

[0209] (5) By enabling content distribution services, usually such as recommendation engines or search engines or operating direct display pages, multimedia objects that have passed the manual review system are provided to the terminal for display.

[0210] (6) Responsible for communicating with the high-order semantic tag mining service to complete the high-order semantic tag mining supplementary processing of multimedia objects.

[0211] 5. Manual Review System

[0212] (1) It is necessary to read the original information of the multimedia object itself in the content database. This is usually a system developed based on a web (network) database with complex business operations. Manual judgment is made on whether the multimedia object involves abnormal information, thereby performing a preliminary filtering.

[0213] (2) Based on the initial review, a secondary review is conducted on the multimedia objects, mainly to classify, label or confirm the labels of the multimedia objects. By conducting a secondary manual review on the results of machine processing, the accuracy and efficiency of the labels corresponding to the multimedia objects can be improved through human-machine collaboration.

[0214] (3) The processing of the high-level semantic tag set mentioned in the embodiments of the present application can be checked and organized with the help of a manual review system.

[0215] 6. Content deduplication service

[0216] (1) Communicate with the dispatch center server. The main tasks include deduplication of titles, cover images, content, and video and audio fingerprints. For graphic content objects, the title and text of the graphic content object are usually vectorized, and the text feature vector is generated and deduplicated using simmhash (text fingerprint) and BERT. The image feature vector is also determined and deduplicated. For video content objects, the video fingerprint and audio fingerprint are extracted to construct the video feature vector. Then, the distance between the video feature vectors is calculated, such as the Euclidean distance, to determine whether the video content is repeated.

[0217] 7. High-level semantic tag dictionary

[0218] (1) According to the relevant steps provided in the above method embodiment, high-order semantic tags are mined based on the semantic text information associated with the information release object, and the above-mentioned high-order semantic tag dictionary library is generated. There are a large number of multimedia object collections (such as video collections, graphic content collections, etc.) in the information flow content service. The semantic text information associated with the information release object corresponding to the multimedia object collection itself can reflect high-order semantic information. The use of semantic text information and multimedia object clustering can assist in the efficiency of high-order semantic tag mining.

[0219] 8. Information Acquisition Services

[0220] (1) According to the relevant steps provided in the above method embodiment, the information acquisition service can obtain the information release object and its associated semantic text information from the network as a data source for summarizing and mining the high-order semantic tags in the high-order semantic tag dictionary library, and also provide a data source for clustering.

[0221] (2) Optionally, the prototype of the information acquisition engine of the information acquisition system corresponding to the information acquisition service can be implemented using Puppeteer. Puppeteer is a Node.js package that simulates the operation of a browser. Node.js is a JavaScript runtime environment based on a browser engine. JavaScript is a lightweight, interpreted or just-in-time compiled programming language with function priority. The API provided by Puppeteer can easily control the browser to implement information acquisition applications, website screenshots, generate website PDFs (Portable Document Format), etc.

[0222] 9. Content Storage Service

[0223] (1) It usually consists of a group of storage servers that are widely distributed and convenient for terminals to access nearby. Usually there are also CDN acceleration servers in the periphery for distributed cache acceleration, which saves the multimedia objects uploaded by content producers through the upstream and downstream content interface services.

[0224] (2) After obtaining the index information, the content consumer can also directly access the content storage server to download the corresponding multimedia object.

[0225] (3) In addition to being a data source for external services, it also serves as a data source for internal services, allowing the download file system to obtain original multimedia object data for related processing. The paths of internal and external data sources are usually deployed separately to avoid mutual influence.

[0226] 10. Advanced Semantic Tag Mining Service

[0227] (1) The high-order semantic tag mining model is made into a service, and communicates with the central server to complete the mining and supplement of high-order semantic tags of multimedia objects on the main link of the process.

[0228] (2) The high-level semantic tag mining service includes two major parts: recall and sorting. For specific functions and processing systems, please refer to the relevant introduction in the above method embodiment.

[0229] 11. High-level semantic label recall and ranking model

[0230] (1) According to the detailed description process of the above method embodiment, the end-to-end classification problem-solving approach is transformed into a matching problem, and high-order semantic labels are mined through the recall + sorting structure.

[0231] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0232] Please refer to Figure 15 , which shows a block diagram of a label determination device provided by one embodiment of the present application. The device has the function of implementing the above-mentioned label determination method, and the function can be implemented by hardware or by hardware executing corresponding software. The device can be a computer device or can be set in a computer device. The device 1500 may include: an object feature acquisition module 1510, a label recall module 1520, a candidate label determination module 1530, and a first label determination module 1540.

[0233] The object feature acquisition module 1510 is configured to acquire a first multimedia object and feature information corresponding to the first multimedia object.

[0234] The label recall module 1520 is used to perform label recall processing based on the feature information to obtain a label recall result; wherein, the label recall result includes a second semantic label corresponding to at least one second multimedia object in a high-order semantic label set, and the second multimedia object is a multimedia object in at least one multimedia object whose similarity with the first multimedia object is higher than a similarity threshold, and the high-order semantic label in the high-order semantic label set is determined based on the semantic text information associated with the information release object corresponding to the at least one multimedia object.

[0235] The candidate tag determination module 1530 is configured to obtain a candidate tag set based on the second semantic tag.

[0236] The first tag determination module 1540 is configured to determine a target candidate tag in the candidate tag set that meets preset conditions as a first semantic tag corresponding to the first multimedia object, where the first semantic tag is used to represent high-order semantic information of the first multimedia object.

[0237] In an exemplary embodiment, the apparatus 1500 further includes: an object clustering module, a semantic text acquisition module, a high-level tag acquisition module, and a tag set generation module.

[0238] The object clustering module is configured to perform clustering processing on the at least one multimedia object to obtain at least one multimedia object set and at least one information publishing object set corresponding to the at least one multimedia object set.

[0239] The semantic text acquisition module is used to acquire semantic text information associated with the information release object in the at least one information release object set.

[0240] The high-level tag acquisition module is used to obtain at least one high-level semantic tag corresponding to the at least one multimedia object set based on the semantic text information.

[0241] The tag set generation module is configured to obtain the high-order semantic tag set based on the at least one high-order semantic tag.

[0242] In an exemplary embodiment, the semantic text acquisition module includes: a description information acquisition unit, a topic information acquisition unit, and a search information acquisition unit.

[0243] The description information acquisition unit is configured to acquire the publishing object description information corresponding to the information publishing object in the at least one information publishing object set when the semantic text information includes publishing object description information.

[0244] The subject information acquisition unit is used to determine the target multimedia object corresponding to the information publishing object when the semantic text information includes content subject information; and acquire the content subject information corresponding to the target multimedia object.

[0245] The search information acquisition unit is configured to acquire the search information associated with the target multimedia object when the semantic text information includes search information.

[0246] In an exemplary embodiment, the feature information includes at least one search term corresponding to the first multimedia object, where the at least one search term is used to represent search intent information corresponding to the first multimedia object. The candidate tag determination module 1530 is further configured to:

[0247] The candidate tag set is obtained based on the second semantic tag and the at least one search term.

[0248] In an exemplary embodiment, the feature information includes entity information and keyword information corresponding to the first multimedia object, and the tag recall module 1520 includes: a first recall unit and a result generation unit.

[0249] The first recall unit is used to compare and recall the entity information and keyword information corresponding to the first multimedia object with the entity information and keyword information corresponding to the at least one multimedia object to obtain a first recall object, and the at least one second multimedia object includes the first recall object.

[0250] A result generating unit is configured to obtain the label recall result based on the high-order semantic label corresponding to the first recalled object in the high-order semantic label set.

[0251] In an exemplary embodiment, the feature information includes a feature vector of at least one dimension corresponding to the first multimedia object, and the tag recall module 1520 further includes: a second recall unit.

[0252] The second recall unit is used to compare and recall the feature vector of at least one dimension corresponding to the first multimedia object with the feature vector of at least one dimension corresponding to the at least one multimedia object to obtain a second recall object, and the at least one second multimedia object includes the second recall object.

[0253] The result generating unit is further configured to obtain the label recall result based on the high-order semantic label corresponding to the second recalled object in the high-order semantic label set.

[0254] In an exemplary embodiment, the first tag determination module 1540 includes: an importance data determination unit, an overall importance determination unit, a tag sorting unit, a tag screening unit, and a tag determination unit.

[0255] An importance data determination unit is used to determine the first importance data corresponding to the candidate tag in the candidate tag set in the high-order semantic tag set, and the second importance data corresponding to the candidate tag in the target tag set, wherein the first importance data is used to characterize the importance of the candidate tag in the high-order semantic tag set, and the second importance data is used to characterize the importance of the candidate tag in the target tag set.

[0256] The overall importance determination unit is used to determine the overall importance data corresponding to the candidate tag based on the first importance data and the second importance data, wherein the overall importance data is used to represent the overall importance of the candidate tag.

[0257] The tag sorting unit is used to sort the candidate tags in descending order according to the overall importance data to obtain a sorting result.

[0258] The tag screening unit is used to determine the first n candidate tags in the arrangement result as the target candidate tags, where n is an integer greater than 0.

[0259] A label determination unit is configured to determine the target candidate label as the first semantic label.

[0260] In an exemplary embodiment, the overall importance determination unit includes: a difference data determination subunit and an overall importance determination subunit.

[0261] The difference data determining subunit is configured to determine importance difference data between the first importance data and the second importance data.

[0262] The overall importance determination subunit is configured to determine the overall importance data based on the importance difference data and the first importance data.

[0263] To sum up, the technical solution provided by the embodiment of the present application, after obtaining the first multimedia object without high-order semantic tags and its feature information, can perform label recall based on the feature information to obtain the label recall result, and use the high-order semantic tag corresponding to the second multimedia object similar to the first multimedia object in the label recall result as the candidate tag, and screen the candidate tags through preset conditions to determine the high-order semantic tag of the first multimedia object. The label marking process does not require human participation, which reduces labor costs, and the determined high-order semantic tag is determined by the semantic text information associated with the information release object corresponding to the multimedia object, which can more accurately reflect the semantic information of the multimedia object, conform to the user's subjective feelings, and improve the accuracy of the information tag.

[0264] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0265] Please refer to Figure 16 , which shows a block diagram of a computer device provided by an embodiment of the present application. The computer device may be a server for executing the above-mentioned tag determination method. Specifically:

[0266] Computer device 1600 includes a central processing unit (CPU) 1601, a system memory 1604 including a random access memory (RAM) 1602 and a read-only memory (ROM) 1603, and a system bus 1605 connecting system memory 1604 and CPU 1601. Computer device 1600 also includes a basic input / output system (I / O system) 1606 that facilitates information transfer between various components within the computer, and a mass storage device 1607 for storing an operating system 1613, application programs 1614, and other program modules 1615.

[0267] The basic input / output system 1606 includes a display 1608 for displaying information and an input device 1609, such as a mouse and keyboard, for user input. Both the display 1608 and the input device 1609 are connected to the central processing unit 1601 via an input / output controller 1610 connected to the system bus 1605. The basic input / output system 1606 may also include an input / output controller 1610 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1610 also provides output to a display screen, printer, or other types of output devices.

[0268] The mass storage device 1607 is connected to the central processing unit 1601 via a mass storage controller (not shown) connected to the system bus 1605. The mass storage device 1607 and its associated computer-readable media provide non-volatile storage for the computer device 1600. In other words, the mass storage device 1607 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0269] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above-mentioned ones. The above-mentioned system memory 1604 and mass storage device 1607 can be collectively referred to as memory.

[0270] According to various embodiments of the present application, the computer device 1600 may also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 1600 may be connected to a network 1612 via a network interface unit 1611 connected to the system bus 1605, or the network interface unit 1611 may be used to connect to other types of networks or remote computer systems (not shown).

[0271] The memory further includes a computer program, which is stored in the memory and configured to be executed by one or more processors to implement the above-mentioned tag determination method.

[0272] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. When the at least one instruction, the at least one program, the code set or the instruction set is executed by a processor, the above-mentioned label determination method is implemented.

[0273] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or an optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0274] In an exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described tag determination method.

[0275] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application do not limit this.

[0276] In addition, in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0277] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A label determination method, characterized in that: The method comprises: Obtaining a first multimedia object and feature information corresponding to the first multimedia object; performing label recall processing based on the feature information to obtain a label recall result; wherein the label recall result includes a second semantic label corresponding to at least one second multimedia object in a high-order semantic label set, the second multimedia object being a multimedia object in at least one multimedia object whose similarity to the first multimedia object is greater than a similarity threshold, at least one high-order semantic label in the high-order semantic label set is determined based on semantic text information associated with an information release object in at least one information release object set, at least one multimedia object set corresponds to at least one high-order semantic label, and the at least one multimedia object set and the at least one information release object set are obtained by clustering the at least one multimedia object; Based on the second semantic label, obtaining a candidate label set; A target candidate tag that meets a preset condition in the candidate tag set is determined as a first semantic tag corresponding to the first multimedia object, where the first semantic tag is used to represent high-order semantic information of the first multimedia object.

2. The method according to claim 1, characterized in that The method further comprises: performing clustering processing on the at least one multimedia object to obtain at least one multimedia object set and at least one information publishing object set corresponding to the at least one multimedia object set; Acquiring semantic text information associated with an information publishing object in the at least one information publishing object set; Based on the semantic text information, obtaining at least one high-order semantic tag corresponding to the at least one multimedia object set; The high-order semantic tag set is obtained based on the at least one high-order semantic tag.

3. The method according to claim 2, characterized in that The obtaining of semantic text information associated with the information release object in the at least one information release object set includes: In a case where the semantic text information includes publishing object description information, obtaining publishing object description information corresponding to the information publishing object in the at least one information publishing object set; In the case where the semantic text information includes content theme information, determining a target multimedia object corresponding to the information publishing object; and obtaining the content theme information corresponding to the target multimedia object; In a case where the semantic text information includes search information, the search information associated with the target multimedia object is obtained.

4. The method according to claim 1, wherein The feature information includes at least one search term corresponding to the first multimedia object, where the at least one search term is used to represent search intent information corresponding to the first multimedia object. The obtaining of a candidate tag set based on the second semantic tag includes: The candidate tag set is obtained based on the second semantic tag and the at least one search term.

5. The method according to claim 1, wherein The feature information includes entity information and keyword information corresponding to the first multimedia object, and performing label recall processing based on the feature information to obtain a label recall result includes: performing a comparison and recall process on the entity information and keyword information corresponding to the first multimedia object and the entity information and keyword information corresponding to the at least one multimedia object to obtain a first recalled object, wherein the at least one second multimedia object includes the first recalled object; The label recall result is obtained based on the high-order semantic label corresponding to the first recalled object in the high-order semantic label set.

6. The method according to claim 1, characterized in that The feature information includes a feature vector of at least one dimension corresponding to the first multimedia object, and performing label recall processing according to the feature information to obtain a label recall result includes: performing a comparison and recall process on a feature vector of at least one dimension corresponding to the first multimedia object and a feature vector of at least one dimension corresponding to the at least one multimedia object to obtain a second recalled object, wherein the at least one second multimedia object includes the second recalled object; The label recall result is obtained based on the high-order semantic label corresponding to the second recalled object in the high-order semantic label set.

7. The method according to any one of claims 1 to 6, characterized in that The step of determining a target candidate tag that meets a preset condition in the candidate tag set as a first semantic tag corresponding to the first multimedia object includes: Determine first importance data corresponding to a candidate tag in the candidate tag set in the high-order semantic tag set, and second importance data corresponding to the candidate tag in the target tag set, wherein the first importance data is used to characterize the importance of the candidate tag in the high-order semantic tag set, and the second importance data is used to characterize the importance of the candidate tag in the target tag set; Determining overall importance data corresponding to the candidate tag based on the first importance data and the second importance data, wherein the overall importance data is used to represent the overall importance of the candidate tag; Arrange the candidate tags in descending order according to the overall importance data to obtain an arrangement result; Determine the first n candidate tags in the arrangement result as the target candidate tag, where n is an integer greater than 0; The target candidate tag is determined as the first semantic tag.

8. The method according to claim 7, characterized in that The determining, based on the first importance data and the second importance data, the overall importance data corresponding to the candidate tag includes: determining importance difference data between the first importance data and the second importance data; The overall importance data is determined based on the importance difference data and the first importance data.

9. A label determination device, characterized in that: The device comprises: an object feature acquisition module, configured to acquire a first multimedia object and feature information corresponding to the first multimedia object; a tag recall module configured to perform tag recall processing based on the feature information to obtain a tag recall result; wherein the tag recall result includes a second semantic tag corresponding to at least one second multimedia object in a high-order semantic tag set, the second multimedia object being a multimedia object in at least one multimedia object whose similarity to the first multimedia object is greater than a similarity threshold, at least one high-order semantic tag in the high-order semantic tag set is determined based on semantic text information associated with an information release object in at least one information release object set, at least one multimedia object set corresponds to at least one high-order semantic tag, and the at least one multimedia object set and the at least one information release object set are obtained by clustering the at least one multimedia object; a candidate tag determination module, configured to obtain a candidate tag set based on the second semantic tag; The first tag determination module is configured to determine a target candidate tag in the candidate tag set that meets preset conditions as a first semantic tag corresponding to the first multimedia object, where the first semantic tag is used to represent high-order semantic information of the first multimedia object.

10. The device according to claim 9, characterized in that The device further comprises: an object clustering module, configured to perform clustering processing on at least one multimedia object to obtain at least one multimedia object set and at least one information publishing object set corresponding to the at least one multimedia object set; A semantic text acquisition module, configured to acquire semantic text information associated with an information release object in the at least one information release object set; A high-level tag acquisition module, configured to obtain at least one high-level semantic tag corresponding to the at least one multimedia object set based on the semantic text information; The tag set generation module is configured to obtain the high-order semantic tag set based on the at least one high-order semantic tag.

11. The device according to claim 10, characterized in that The semantic text acquisition module includes: a description information acquisition unit, configured to acquire, when the semantic text information includes the release object description information, the release object description information corresponding to the information release object in the at least one information release object set; a subject information acquisition unit configured to determine a target multimedia object corresponding to the information publishing object when the semantic text information includes content subject information; and acquire content subject information corresponding to the target multimedia object; The search information acquisition unit is configured to acquire the search information associated with the target multimedia object when the semantic text information includes search information.

12. The device according to claim 9, characterized in that The feature information includes at least one search term corresponding to the first multimedia object, where the at least one search term is used to represent search intent information corresponding to the first multimedia object. The candidate tag determination module is further configured to: The candidate tag set is obtained based on the second semantic tag and the at least one search term.

13. The device according to claim 9, characterized in that The feature information includes entity information and keyword information corresponding to the first multimedia object, and the tag recall module includes: a first recall unit, configured to compare and recall the entity information and keyword information corresponding to the first multimedia object with the entity information and keyword information corresponding to the at least one multimedia object, to obtain a first recalled object, wherein the at least one second multimedia object includes the first recalled object; A result generating unit is configured to obtain the label recall result based on the high-order semantic label corresponding to the first recalled object in the high-order semantic label set.

14. The device according to claim 13, characterized in that The feature information includes a feature vector of at least one dimension corresponding to the first multimedia object, and the tag recall module further includes: a second recall unit, configured to perform a comparison and recall process on the feature vector of at least one dimension corresponding to the first multimedia object and the feature vector of at least one dimension corresponding to the at least one multimedia object, to obtain a second recalled object, wherein the at least one second multimedia object includes the second recalled object; The result generating unit is further configured to obtain the label recall result based on the high-order semantic label corresponding to the second recalled object in the high-order semantic label set.

15. The device according to any one of claims 9 to 14, characterized in that The first tag determination module includes: an importance data determining unit, configured to determine first importance data corresponding to a candidate tag in the candidate tag set in the high-order semantic tag set, and second importance data corresponding to the candidate tag in the target tag set, wherein the first importance data is used to characterize the importance of the candidate tag in the high-order semantic tag set, and the second importance data is used to characterize the importance of the candidate tag in the target tag set; an overall importance determination unit, configured to determine overall importance data corresponding to the candidate tag based on the first importance data and the second importance data, wherein the overall importance data is used to represent the overall importance of the candidate tag; a tag sorting unit, configured to sort the candidate tags in descending order according to the overall importance data to obtain a sorting result; a label screening unit, configured to determine the first n candidate labels in the arrangement result as the target candidate label, where n is an integer greater than 0; A label determination unit is configured to determine the target candidate label as the first semantic label.

16. The device according to claim 15, characterized in that The overall importance determination unit includes: a difference data determining subunit, configured to determine importance difference data between the first importance data and the second importance data; The overall importance determination subunit is configured to determine the overall importance data based on the importance difference data and the first importance data.

17. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the label determination method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the label determination method according to any one of claims 1 to 8.

19. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes to implement the label determination method according to any one of claims 1 to 8.

Images