Visual style network training methods, visual style recognition methods, devices and equipment

Abstract

This application provides a visual style network training method, a visual style recognition method, an apparatus, and a device, applicable to various scenarios such as cloud technology, artificial intelligence, mapping, intelligent transportation, assisted driving, and in-vehicle systems. The visual style network training method includes: acquiring at least two sample data pairs, and object interaction data corresponding to each data point in each sample data pair; determining initial sample data pairs; acquiring target candidate objects that interact with each data point in the initial sample data pairs; determining target sample data pairs based on the object attribute information of the target candidate objects; and constructing positive sample data, negative sample data, and anchor target sample data for training the visual style network based on the target sample data pairs. This application's embodiments improve the training accuracy and efficiency of the visual style network by combining unsupervised and weakly supervised methods. Using the visual style network, label recognition of all visual styles can be performed, thus expanding the recommendation range.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a visual style network training method, a visual style recognition method, an apparatus and device. Background Technology

[0002] In information flow recommendation systems, existing content classification and tagging systems cannot meet the diverse needs of end users, nor can they accurately control the overall visual style. Therefore, it is necessary to characterize content from the perspective of visual style (e.g., understated, youthful). Visual style tags are then used as a recommendation feature, combined with recommendation-side recall and ranking, to achieve accurate recommendations. Visual style can refer to the title of the content consumed by the end user (e.g., a user) in the information flow, or the intuitive feeling of the content's cover image and author. It may reflect the content's emotional expression (e.g., exaggerated style), the direct emotional feeling the content evokes in the end user (e.g., understated, youthful), or the content creator's creative style.

[0003] In related technologies, the method for characterizing visual style is to label visual style data and then train a supervised model based on the labeled visual style data. However, the labeled data mainly focuses on high-priority tags in the current content scenario, and cannot label all visual styles existing in the content data, thus failing to cover a more comprehensive combination of tags, thereby reducing the recommendation scope and accuracy. In addition, the labeling standards are highly subjective and the labeling speed is slow, which reduces recommendation accuracy and efficiency, and increases the consumption of system resources in the recommendation process. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a visual style network training method, a visual style recognition method, an apparatus, and a device.

[0005] On the one hand, this application proposes a method for training a visual style network, the method comprising:

[0006] Obtain at least two sample data pairs, and the object interaction data corresponding to each data in each sample data pair; the display data of each data in each sample data pair is the same, and the content of each data in each sample data pair matches.

[0007] Based on the object interaction data, an initial sample data pair is determined from the at least two sample data pairs; the initial sample data pair includes an initial positive sample data pair and an initial negative sample data pair; the object interaction data corresponding to each data in the initial positive sample data pair satisfies a first preset condition; the object interaction data corresponding to each data in the initial negative sample data pair satisfies a second preset condition.

[0008] Show each data contained in the initial sample data pair to the candidate object set, and obtain target candidate objects that interact with each data contained in the initial sample data pair from the candidate object set;

[0009] Based on the object attribute information of the target candidate object, target sample data pairs are determined from the initial sample data pairs; the target sample data pairs include target positive sample data pairs determined from the initial positive sample data pairs and target negative sample data pairs determined from the initial negative sample data pairs, wherein the visual styles of each data in the target positive sample data pairs match, and the visual styles of each data in the target negative sample data pairs do not match.

[0010] Based on the target sample data pairs, positive sample data, negative sample data, and anchored target sample data are constructed; the content of the positive sample data matches the content of the anchored target sample data, and the content of the negative sample data does not match the content of the anchored target sample data.

[0011] Based on the differences between the positive sample data and the anchored target sample data, and the differences between the negative sample data and the anchored target sample data, loss data is calculated, and a preset network is trained based on the loss data to obtain a visual style network.

[0012] On the other hand, this application proposes a visual style recognition method, which includes:

[0013] Obtain at least two data items to be processed;

[0014] The at least two data to be processed are input into a visual style network for visual style recognition to obtain the visual style feature vectors corresponding to each of the at least two data to be processed.

[0015] Cluster the visual style feature vectors corresponding to each of the at least two data to be processed to obtain the visual style feature clustering results corresponding to the at least two data to be processed.

[0016] The visual style feature clustering results are sent to the client so that the client can make information recommendations based on the visual style feature clustering results;

[0017] The visual style network is trained using the visual style network training method described above.

[0018] On the other hand, this application proposes a training device for a visual style network, the device comprising:

[0019] The sample data pair and interactive data acquisition module is used to acquire at least two sample data pairs, as well as the object interactive data corresponding to each data in each sample data pair; the display data of each data in each sample data pair is the same, and the content of each data in each sample data pair matches.

[0020] An initial sample data pair determination module is used to determine an initial sample data pair from at least two sample data pairs based on the object interaction data; the initial sample data pair includes an initial positive sample data pair and an initial negative sample data pair; the object interaction data corresponding to each data in the initial positive sample data pair satisfies a first preset condition; the object interaction data corresponding to each data in the initial negative sample data pair satisfies a second preset condition.

[0021] The target candidate object acquisition module is used to display each data contained in the initial sample data pair to the candidate object set, and to acquire target candidate objects that interact with each data contained in the initial sample data pair from the candidate object set.

[0022] The target sample data pair determination module is used to determine target sample data pairs from the initial sample data pairs based on the object attribute information of the target candidate objects; the target sample data pairs include target positive sample data pairs determined from the initial positive sample data pairs and target negative sample data pairs determined from the initial negative sample data pairs, wherein the visual styles of the data in the target positive sample data pairs match, and the visual styles of the data in the target negative sample data pairs do not match.

[0023] A construction module is used to construct positive sample data, negative sample data, and anchored target sample data based on the target sample data pair; the content of the positive sample data matches the content of the anchored target sample data, and the content of the negative sample data does not match the content of the anchored target sample data.

[0024] The training module is used to calculate loss data based on the difference between the positive sample data and the anchored target sample data, and the difference between the negative sample data and the anchored target sample data, and to train a preset network based on the loss data to obtain a visual style network.

[0025] On the other hand, embodiments of this application provide a visual style recognition device, the device comprising:

[0026] The data acquisition module is used to acquire at least two data items to be processed.

[0027] The recognition module is used to input the at least two data to be processed into the visual style network to perform visual style recognition and obtain the visual style feature vectors corresponding to each of the at least two data to be processed.

[0028] The clustering module is used to perform clustering processing on the visual style feature vectors corresponding to each of the at least two data to be processed, so as to obtain the visual style feature clustering results corresponding to the at least two data to be processed.

[0029] The sending module is used to send the style feature clustering results to the client so that the client can make information recommendations based on the visual style feature clustering results; wherein, the visual style network is trained using the above-mentioned visual style network training method.

[0030] On the other hand, this application proposes an electronic device for visual style network training or visual style recognition, the electronic device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the visual style network training method or visual style recognition method as described above.

[0031] On the other hand, this application proposes a computer-readable storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement the visual style network training method or visual style recognition method as described above.

[0032] On the other hand, this application proposes a computer program product that, when executed by a processor, implements the visual style network training method or visual style recognition method as described above.

[0033] The visual style network training, visual style recognition method, apparatus, and device proposed in the embodiments of this application are specifically for the visual style network training method:

[0034] Acquire at least two sample data pairs, and object interaction data corresponding to each data point in each sample data pair; the display data of each data point in each sample data pair is the same, and the content of each data point in each sample data pair matches; based on the object interaction data, determine initial sample data pairs from the at least two sample data pairs, including initial positive sample data pairs and initial negative sample data pairs; display each data point in the initial sample data pairs to a candidate object set, and obtain target candidate objects that interact with each data point in the initial sample data pairs from the candidate object set; determine target sample data pairs from the initial sample data pairs based on the object attribute information of the target candidate objects; construct positive sample data, negative sample data, and anchored target sample data based on the target sample data pairs; train a preset network based on the positive sample data, negative sample data, and anchored target sample data to obtain a visual style network. The process of determining the initial sample data pairs is essentially an unsupervised process, while the process of determining the target sample data pairs is essentially a weakly supervised process. Since the data displayed is identical and the content matches, the differences in the corresponding object interaction data can serve as a basis for determining whether the objects' emotional feelings towards the data are consistent or whether the creative styles of the data are consistent. Furthermore, since the object attribute information of the target candidate objects is consistent, it can further serve as a basis for determining whether the objects' emotional feelings towards the data are consistent or whether the creative styles of the data are consistent. Therefore, by combining unsupervised and weakly supervised methods, a visual style feature space is constructed, stripping away the content features themselves. This allows the vectors extracted by the network to represent features that focus more on visual style, enabling the network to learn the differences in visual style and thus improving the training accuracy of the visual style network. Since it does not require labeled data, it improves the efficiency of network training and reduces the cost of network training.

[0035] For the visual style recognition method: At least two pieces of data to be processed are input into the visual style network for visual style recognition, resulting in visual style feature vectors corresponding to each of the at least two pieces of data. These visual style feature vectors are then clustered to obtain a visual style feature clustering result. Since the clustered visual style feature result can label all visual styles present in the content data, thus covering a more comprehensive range of tags, this clustered result is sent to the client. The client then uses this result to make information recommendations, thereby improving the recommendation scope and accuracy while reducing the consumption of system resources during the recommendation process. Attached Figure Description

[0036] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram illustrating the implementation environment of a visual style network training method and a visual style recognition method according to an exemplary embodiment.

[0038] Figure 2 This is a flowchart illustrating a visual style network training method according to an exemplary embodiment. Figure 1 .

[0039] Figure 3 This is a flowchart illustrating a visual style network training method according to an exemplary embodiment. Figure 2 .

[0040] Figure 4 This is a schematic diagram illustrating the determination of target negative sample data pairs according to an exemplary embodiment.

[0041] Figure 5 This is a flowchart illustrating a visual style network training method according to an exemplary embodiment. Figure 3 .

[0042] Figure 6 This is a schematic diagram of a preset network structure according to an exemplary embodiment.

[0043] Figure 7 This is a schematic diagram of another preset network structure according to an exemplary embodiment.

[0044] Figure 8 This is a flowchart illustrating a visual style recognition method according to an exemplary embodiment.

[0045] Figure 9 This is a schematic diagram illustrating the principle of the Kmeans elbow rule according to an exemplary embodiment.

[0046] Figure 10 This is a schematic diagram illustrating a visual style feature clustering result according to an exemplary embodiment.

[0047] Figure 11 This is a schematic diagram of a system for visual style recognition based on a combination of unsupervised and weakly supervised visual style network training, according to an exemplary embodiment.

[0048] Figure 12This is a block diagram of a visual style network training device according to an exemplary embodiment.

[0049] Figure 13 This is a block diagram illustrating a visual style recognition device according to an exemplary embodiment.

[0050] Figure 14 This is a hardware structure block diagram of a server for visual style network training or visual style recognition, provided according to an exemplary embodiment. Detailed Implementation

[0051] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0052] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning. Machine learning, in particular, is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0053] Specifically, the training process of the visual style network in the embodiments of this application involves machine learning techniques.

[0054] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0055] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0056] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0057] The following describes the technical terms used in the embodiments of this application:

[0058] Professionally Generated Content (PGC): This term broadly refers to content personalization, diversified perspectives, and virtualized social relationships.

[0059] User-generated content (UGC) refers to content created by users, that is, content that users create and display or provide to other users through internet platforms.

[0060] Multi-Channel Network (MCN): This is a product form that combines PGC content and, with strong capital support, ensures the continuous output of content, thereby ultimately achieving stable commercial monetization.

[0061] Feeds: Also translated as sources, feeds, information providers, news feeds, summaries, sources, content subscriptions, web sources, etc. Feeds are a data format through which websites disseminate the latest information to users, usually arranged in a timeline. A prerequisite for users to subscribe to a website is that the website provides news sources. The aggregation of feeds in one place is called aggregation, and the software used for aggregation is called an aggregator.

[0062] Videos: Videos recommended to users in the feed, including vertical short videos and horizontal short videos, are provided in the form of a feed stream.

[0063] Creative style refers to the style of drawing. It refers to the brushstrokes and composition used in drawing, expressing the same artistic conception in different styles, highlighting one's understanding on the canvas through artistic means, and leaving different feelings in the viewer. In the art world (including comics), creative style refers to the drawing style, including the artist's techniques for drawing characters, scenery, etc., and the handling of lines, such as the lines used for the nose, eyes, ears, muscles, and hair. In illustration and other colored paintings, it also includes the handling of color, tone, and other aspects.

[0064] Embedding refers to a method of converting discrete variables into continuous vectors.

[0065] Currently, news feeds and short videos are two sub-sectors maintaining strong growth in the user domain. News feed products can provide users with suitable content in appropriate scenarios. The essence of news feed products is content recommendation, which can be achieved through two methods: one is manual operation, where people use their experience to judge the value of news; the other relies on machines and algorithms for recommendation. Each method has its advantages and disadvantages. Short videos refer to frequently pushed video content, ranging from a few seconds to a few minutes, played on various new media platforms, suitable for viewing on mobile devices and during short leisure periods. The content integrates themes such as skill sharing, humor, fashion trends, social hot topics, street interviews, public welfare education, advertising creativity, and commercial customization. Due to their short length, they can be standalone or part of a series. From its initial UGC, PGC, and user uploads, to specialized short video production agencies, MCNs, and professional short video applications (Apps), numerous leading traffic platforms have emerged, making short videos a crucial means of content creation and social media platform dissemination. Both producers and consumers of short video content have become a huge group.

[0066] Visual style distinguishes content from existing classification and tagging systems, offering a different dimension for content characterization. For example, entertainment content might be categorized and tagged as gossip, celebrities, or drama trailers, but its visual style can vary, offering different experiences like lighthearted entertainment or understated entertainment. The same understated visual style might be found in different classification systems, such as emotional programs or dramas. Theoretically, classification or tagging systems are orthogonal to visual style systems. To improve recommendation accuracy and scope, content characteristics need to be described from another dimension. Any content that provides a unique user experience can be defined as having its own unique visual style characteristics. Based on this, to achieve accurate recommendations for information streams (e.g., short video content), this application provides a visual style network training method, a visual style recognition method, an apparatus, and a device.

[0067] Figure 1 This is a schematic diagram illustrating the implementation environment of a visual style network training method and a visual style recognition method according to an exemplary embodiment. For example... Figure 1 As shown, the implementation environment may include at least a client 01 and a server 02. The client 01 and the server 02 may be directly or indirectly connected through wired or wireless communication. This embodiment of the application does not impose any limitations on this.

[0068] Specifically, the server 02 can be used to acquire at least two sample data pairs, and object interaction data corresponding to each data in each sample data pair; the display data of each data in each sample data pair is the same, and the content of each data in each sample data pair matches; and to determine an initial sample data pair from at least two sample data pairs based on the object interaction data; and to display each data in the initial sample data pair to a candidate object set, and to acquire target candidate objects from the candidate object set that generate object interaction data with each data in the initial sample data pair; and to determine a target sample data pair from the initial sample data pairs based on the object attribute information of the target candidate objects; and to construct positive sample data, negative sample data, and anchored target sample data based on the target sample data pair; and to train a preset network based on the positive sample data, negative sample data, and anchored target sample data to obtain a visual style network. The system includes: acquiring at least two data sets to be processed; inputting the at least two data sets to be processed into a visual style network for visual style recognition to obtain visual style feature vectors corresponding to each of the at least two data sets; clustering the visual style feature vectors corresponding to each of the at least two data sets to obtain visual style feature clustering results corresponding to the at least two data sets; and sending the style feature clustering results to the client. Optionally, the server 02 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0069] Specifically, the client 01 can be used for information recommendation based on visual style feature clustering results. Optionally, the client 01 can be, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. This application embodiment can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0070] It should be noted that, Figure 1 This is merely one example. Other implementation environments can be included in other scenarios. For instance, the application could be used in an environment that includes a terminal, which could include a network training module to train a pre-defined network to obtain the interest processing network.

[0071] In information flow recommendation, since the existing content classification and tagging system cannot control the overall visual style, this application proposes an abstract semantic description of visual style multimodal to characterize the first emotional feeling of the client when seeing the content, and to identify the content from a higher dimension. The typical tags that specifically characterize the content may include, but are not limited to: exaggerated emotions, rural style, light entertainment, low-key, easy to understand, serious and formal, high-profile, positive social energy, in-depth professionalism, healing, etc.

[0072] Optionally, this application employs an algorithm combining weakly supervised and unsupervised clustering to address how to construct an abstract and effective pseudo-label recognition system for semantically understanding content in the absence of labeled data, thereby assisting the recommendation system in adjusting the visual style of content distribution. The core idea is: for all content corpora used in information flow distribution, a combination of weakly supervised and unsupervised methods is used to mine unsupervised model corpora suitable for visual styles, constructing triplet pairs of positive sample data (positive-pos), negative sample data (negative-neg), and anchor target sample data (anchor-anchor), and training a visual style network based on these triplet pairs. The core objective of the unsupervised model algorithm in this application is to construct a visual style feature space in the visual style dimension of content consumption, stripping away the content's own content features and extracting the visual style embedding of the content as an intermediate product for downstream content use. Then, based on the trained visual style network, visual style vectors (Embedding) are extracted from all enabled content. The visual style vectors (Embedding) are then clustered and the cluster termination conditions are adjusted to obtain cluster pseudo-labels. The client can use these cluster pseudo-labels as features for content recommendation and distribution.

[0073] The training process of the Visual Style Network is described below:

[0074] Figure 2 This is a flowchart illustrating a visual style network training method according to an exemplary embodiment. Figure 1 This method can be used for Figure 1 In the implementation environment described herein, the steps of the methods described in the embodiments or flowcharts are provided. However, based on conventional or non-inventive labor, more or fewer steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown in the embodiments or drawings... Figure 2 As shown, the method may include:

[0075] S101. Obtain at least two sample data pairs, and the object interaction data corresponding to each data in each sample data pair; the display data of each data in each sample data pair is the same, and the content of each data in each sample data pair matches.

[0076] In this embodiment, collecting suitable corpus for a visual style network is crucial for obtaining the visual style network. However, aside from manual annotation (which is extremely costly and inefficient), there are no clear style signals in the information flow business data. Therefore, when constructing unsupervised corpus, other data performance indicators or metrics can be used for corpus mining. For example, a combination of unsupervised and weakly supervised methods can be used to construct training corpus.

[0077] First, the unsupervised process is introduced: Optionally, at least two sample data pairs in step S101 above are data pairs from all content corpora enabled for information flow distribution, i.e., data pairs from the content data pool. These two sample data pairs can be at least one of video, audio / video, image, and text, and can be represented in the form of Feeds.

[0078] For example, display data can refer to impressions, that is, the number of times the data content is pushed to a client object and seen by the client object. Consistent display data can mean that the impressions of all data points are consistent. For instance, obtaining the impressions within a preset time (e.g., one week) after content publication from a content data pool ensures that each sample data pair has the same exposure time and consistent impressions.

[0079] For example, matching the content of various data can mean that the content contained in each data is similar. The similarity of the content contained in various data can be determined by the following methods: 1) The Term Frequency-Inverse Document Frequency (TF-IDF) similarity of the title keywords of the sample data pair must be higher than a preset similarity threshold; 2) The intersection-union ratio (IU) of the tags of the sample data pair must be higher than a preset IU threshold. The title keywords can be extracted by the server from the sample data pair using Name Entity Recognition (NER), and the tags can come from machine or manual labeling of the content of the sample data pair itself. If the data contained in a sample data pair meet the above two conditions, then the content of the data contained in that sample data pair is considered similar.

[0080] For example, object interaction data can refer to: data obtained by the interaction between the sample data pair and the client object within a certain time period. For example, the click rate of the individual data in each sample data pair being clicked by the object, and / or, voice interaction data of the individual data in each sample data pair being interacted with by the client object via voice.

[0081] In summary, assuming a pair of similar content items, if the client object's emotional response to them is inconsistent, it will result in differences in relative click-through rate consumption, that is, differences in the client object's interaction data. Based on this assumption, in step S101 above, the server can mine data pairs with consistent exposure and similar content from the content pool data to obtain at least two sample data pairs. Then, it obtains the data contained in each sample data pair and interacts with the client object within a certain time period to obtain the object interaction data.

[0082] S103. Based on the object interaction data, determine an initial sample data pair from at least two sample data pairs; the initial sample data pair includes an initial positive sample data pair and an initial negative sample data pair; the object interaction data corresponding to each data in the initial positive sample data pair satisfies a first preset condition; the object interaction data corresponding to each data in the initial negative sample data pair satisfies a second preset condition.

[0083] Optionally, step S103 above is also an unsupervised process. In step S103, the server can define initial positive sample data pairs and initial negative sample data pairs based on the differences in object interaction data corresponding to each data in each sample data pair within a certain time period, thereby constructing the unsupervised triples required by the visual style network. Specifically, the object interaction data corresponding to each data in the initial positive sample data pair satisfies a first preset condition; the object interaction data corresponding to each data in the initial negative sample data pair satisfies a second preset condition. The first and second preset conditions can be set according to actual business needs, and this embodiment does not impose specific limitations here. As an example, the first preset condition can be a condition representing that the visual styles of each data in the initial positive sample data pair match, and the second preset condition can be a condition representing that the visual styles of each data in the initial negative sample data pair do not match. Furthermore, the first preset condition can be a condition that the object interaction data corresponding to each data in the initial positive sample data pair matches; and the second preset condition can be a condition that the difference between the object interaction data corresponding to each data in the initial positive sample data pair is greater than a preset difference threshold.

[0084] In one feasible embodiment Figure 3This is a flowchart illustrating a visual style network training method according to an exemplary embodiment. Figure 2 ,like Figure 3 As shown, determining the initial sample data pair from at least two sample data pairs based on object interaction data can include:

[0085] S1031. Based on object interaction data, determine an initial positive sample data pair and an initial negative sample data pair from at least two sample data pairs; the object interaction data corresponding to each data in the initial positive sample data pair are matched; the difference between the object interaction data corresponding to each data in the initial negative sample data pair is greater than a preset difference threshold.

[0086] S1033. Determine the initial positive sample data pairs and the initial negative sample data pairs as the initial sample data pairs.

[0087] Optionally, in step S1031 above, if the object interaction data corresponding to each data item in a sample data pair are similar, the server considers the object interaction data corresponding to each data item in the sample data pair to satisfy a first preset condition and designates the sample data pair as an initial positive sample data pair. If the difference between the object interaction data corresponding to each data item in a sample data pair is greater than a preset difference threshold, the server considers the object interaction data corresponding to each data item in the sample data pair to satisfy a second preset condition and designates the sample data pair as an initial negative sample data pair.

[0088] Optionally, in step S1033 above, the server can directly determine the initial positive sample data pair and the initial negative sample data pair as the initial sample data pair. Alternatively, the server can further filter the initial positive sample data pair and the initial negative sample data pair according to preset rules to obtain the initial sample data pair. For example, deleting sample data pairs containing uncommon words or obscure terms yields the initial sample data pair.

[0089] It should be noted that the preset difference threshold can be configured according to actual business needs, and no specific restrictions are made here.

[0090] For example, taking click-through rate (CTR) as an example of object interaction data, the server can use data pairs with similar CTRs to the object clicks as initial positive sample data pairs, and data pairs with CTRs differing by more than a factor of 2 as initial negative sample data pairs. Unsupervised triples can then be constructed using these initial negative sample data pairs. Table 1 shows the unsupervised triples constructed in this way, which can include initial positive sample data, initial negative sample data, and initial anchor sample data.

[0091] Table 1 Unsupervised Triplets

[0092]

[0093] In this embodiment, if a pair of similar content items has inconsistent emotional responses from the client, it will result in differences in relative click metrics, i.e., differences in client interaction data. By using pairs of matching object interaction data corresponding to each data item as initial positive sample data pairs, and pairs of object interaction data with differences exceeding a preset threshold as initial negative sample data pairs, an unsupervised Triplet training corpus is constructed. This achieves the creation of a visual style feature space in the visual style dimension of content consumption, stripping away the content's inherent features. The vectors extracted by the network represent features that focus more on visual style, enabling the network to learn differences in visual style and thus improving the training accuracy of the visual style network.

[0094] S105. Present the individual data contained in the initial sample data pair to the candidate object set, and retrieve the target candidate objects that interact with the individual data contained in the initial sample data pair from the candidate object set.

[0095] S107. Based on the object attribute information of the target candidate objects, determine the target sample data pairs from the initial sample data pairs. The target sample data pairs include the target positive sample data pairs determined from the initial positive sample data pairs and the target negative sample data pairs determined from the initial negative sample data pairs. The visual styles of the data in the target positive sample data pairs match, while the visual styles of the data in the target negative sample data pairs do not match.

[0096] In this embodiment of the application, in order to enable the network to learn the differences in visual style, the vector representation focuses more on the features of visual style. After the server obtains the initial sample data pair (i.e., obtains the unsupervised Triple t training corpus), it can use a weakly supervised approach to constrain the unsupervised corpus constructed by the above unsupervised approach through the performance data of the online interaction of the content consumption client object, so as to obtain the target sample data pair.

[0097] For example, the candidate object set can be a group of client objects, and it may include at least two candidate objects. For example, a group of university students, a group of women, a group of men, etc. In step S105 above, the server can expose each piece of data contained in the initial sample data pair to the candidate objects in the candidate object set. Among the candidate objects, there are objects that interact with the data in the initial sample data pair, and there are also objects that do not interact with the data in the initial sample data pair. The server can obtain the target candidate objects that interact with each piece of data contained in the initial sample data pair from the candidate object set.

[0098] In one feasible embodiment, the number of initial sample data pairs is at least two, each initial sample data pair includes first data and second data, and the target candidate objects include a first target candidate object that interacts with the first data and a second target candidate object that interacts with the second data, then continue as follows. Figure 3 As shown, in step S107 above, a weakly supervised approach is used to determine the target sample data pair from the initial sample data pair based on the object attribute information of the target candidate object. This may include:

[0099] S1071. Obtain the first object attribute information of the first target candidate object and the second object attribute information of the second target candidate object.

[0100] S1073. Determine the crossover ratio of object attribute information between the first object attribute information and the second object attribute information.

[0101] S1075. Initial positive sample data pairs whose object attribute information intersection-union ratio is greater than a preset intersection-union ratio threshold are determined as target positive sample data pairs, and initial negative sample data pairs whose object attribute information intersection-union ratio is less than or equal to a preset intersection-union ratio threshold are determined as target negative sample data pairs.

[0102] S1077. Determine the target positive sample data pairs and the target negative sample data pairs as target sample data pairs.

[0103] Optionally, in step S1071 above, each initial sample data pair may include first data and second data. The server may acquire a first target candidate object that interacts with the first data and a second target candidate object that interacts with the second data, and acquire first object attribute information of the first target candidate object and second object attribute information of the second target candidate object. For example, the first and second object attribute information may be object interest data, which may refer to the client object's preferences and interests in the content. The "interaction" may refer to voice interaction or click interaction, etc.

[0104] Optionally, in steps S1073-S1077 above, the server can calculate the intersection-union ratio (IU / R) of object attribute information between the first object attribute information and the second object attribute information. This IU / R can refer to the ratio of the intersection to the union of object attribute information. A preset IU / R threshold is set. If the server determines that the IU / R is greater than the preset threshold, it considers the client objects interacting with the first data and the client objects interacting with the second data to be relatively consistent, meaning the visual styles of the first and second data are considered consistent. In this case, the server can determine the corresponding initial positive sample data pair as the target positive sample data pair. If the server determines that the IU / R is less than or equal to the preset threshold, it considers the client objects interacting with the first data and the client objects interacting with the second data to be significantly different, meaning the visual styles of the first and second data are considered inconsistent. In this case, the server can determine the corresponding initial negative sample data pair as the target negative sample data pair. Finally, the server determines the target positive sample data pair and the target negative sample data pair obtained through weak supervision as the target sample data pair.

[0105] Figure 4 This is a schematic diagram illustrating the determination of target negative sample data pairs according to an exemplary embodiment, such as... Figure 4 As shown, taking click-based interaction as an example, suppose an initial sample data pair is a pair of news articles, including content A and B. The content of A and B is similar, and both are exposed to a target group (i.e., a candidate object set). Within this target group, there are four scenarios: objects that clicked A, objects that clicked B, objects that did not click either A or B, or objects that clicked both A and B. If the objects that clicked A and B are relatively consistent (i.e., the intersection-union ratio of their attribute information is greater than a preset threshold), then A and B are considered to have the same visual style. In the case that this initial sample data pair is an initial positive sample data pair, A and B are taken as the target positive sample data pair. That is, if the unsupervised initial sample data pair is an initial positive sample data pair, and the objects that clicked A and B are relatively consistent, then this initial positive sample data pair is considered the target positive sample data pair. If the object clicked on A is significantly different from the object clicked on B (i.e., the intersection-union ratio of the object attribute information of the object clicked on A and the object clicked on B is less than or equal to the preset intersection-union ratio threshold), then A and B are considered to have inconsistent visual styles. If the initial sample data pair is an initial positive sample data pair, then A and B will not be considered as the target negative sample data pair. In other words, if the initial sample data pair obtained without supervision is an initial positive sample data pair, and the object clicked on A is inconsistent with the object clicked on B, then the initial positive sample data pair is considered not to be the target positive sample data pair.

[0106] If the object clicked on (A) differs significantly from the object clicked on (B) (i.e., the intersection-union ratio of the object attributes of objects A and B is less than or equal to a preset intersection-union threshold), then A and B are considered to have inconsistent visual styles. In this case, if the initial sample data pair is an initial negative sample data pair, A and B are considered as the target negative sample data pair. Conversely, if the unsupervised initial sample data pair is an initial negative sample data pair, and the object clicked on (A) is inconsistent with the object clicked on (B), then the initial negative sample data pair is considered as the target negative sample data pair. If the object clicked on (A) is relatively consistent with the object clicked on (B) (i.e., the intersection-union ratio of the object attributes of objects A and B is greater than a preset intersection-union threshold), then A and B are considered to have consistent visual styles. In this case, if the initial sample data pair is an initial negative sample data pair, A and B are considered as the target negative sample data pair. Conversely, if the unsupervised initial sample data pair is an initial negative sample data pair, and the object clicked on (A) is relatively consistent with the object clicked on (B), then the initial negative sample data pair is not considered as the target negative sample data pair.

[0107] In this embodiment, the unsupervised corpus (initial sample data) constructed using the online interaction performance data of content consumption client objects is constrained by the aforementioned unsupervised method. During the constraint process, the intersection-union ratio (IUU) of the attribute information of the client objects in the interaction data is used to measure the consistency rate of the client object interaction data. Since the weakly supervised mining of client object interactions is subject to indirect client object interaction constraints, coupled with the conditions of content consistency and click differences, a visual style feature space can be further constructed by combining unsupervised and weakly supervised methods. This strips away the content features themselves, allowing the vectors extracted by the network to represent features that focus more on visual style. As a result, the network can further learn the differences in visual style, thereby further improving the training accuracy of the visual style network. In addition, the aforementioned combination of unsupervised and weakly supervised methods does not require labeled data, which improves the efficiency of network training and reduces the cost of network training.

[0108] S109. Based on the target sample data pairs, construct positive sample data, negative sample data, and anchored target sample data. The content of the positive sample data matches the content of the anchored target sample data, while the content of the negative sample data does not match the content of the anchored target sample data.

[0109] Optionally, after obtaining the target sample data pair, the server can construct positive sample data, negative sample data, and anchored target sample data based on the target sample data pair. The content of the positive sample data matches the content of the anchored target sample data, while the content of the negative sample data does not match the content of the anchored target sample data.

[0110] In an exemplary embodiment, in step S109 above, constructing positive sample data, negative sample data, and anchored target sample data based on the target sample data pairs may include:

[0111] Determine any one of the data points in the target positive sample data pair as the anchor target sample data.

[0112] In the target positive sample data pair, the data whose content matches the content of the anchor target sample data is used as the positive sample data.

[0113] In the target negative sample data pair, data whose content does not match the content of the anchored target sample data are identified as negative sample data.

[0114] Optionally, for each target positive sample data pair, the server can use any one data point from the target positive sample data pair as the anchor target sample data. The server then uses data points from the target positive sample data pair whose content matches (e.g., is similar to) the content of the anchor target sample data as positive sample data. Since the two data points in the target sample data are themselves data pairs with similar content, the positive sample data can also be any data point from the target positive sample data pair other than the anchor target sample data. After obtaining the anchor target sample data, the server can randomly select a data point from the target negative sample data pair whose content does not match the content of the anchor target sample data as the negative sample data. This constructs a triplet consisting of positive sample data, negative sample data, and anchor target sample data.

[0115] Since the target sample data pair is constructed based on the above-mentioned combination of unsupervised and weakly supervised methods, a triplet consisting of sample data, negative sample data, and anchored target sample data is further constructed using this target sample data pair. This triplet is also constructed based on the combination of unsupervised and weakly supervised methods. That is, the triplet constructed in this way can remove the content features of the content itself, so that the vector extracted by the network represents features that focus more on visual style. This allows the network to further learn the differences in visual style, thereby further improving the training accuracy of the visual style network.

[0116] S1011. Based on the difference between positive sample data and the anchored target sample data, and the difference between negative sample data and the anchored target sample data, calculate the loss data, train the preset network based on the loss data, and obtain the visual style network.

[0117] In this embodiment of the application, the server can train a preset network based on positive sample data, negative sample data and anchored target sample data to obtain a visual style network.

[0118] In an optional embodiment, Figure 5This is a flowchart illustrating a visual style network training method according to an exemplary embodiment. Figure 3 ,like Figure 5 As shown, in step S1011 above, the calculation of loss data based on the difference between positive sample data and anchored target sample data, and the difference between negative sample data and anchored target sample data, and the training of a preset network based on the loss data to obtain a visual style network, may include:

[0119] S10111. Input the positive sample data, negative sample data, and anchored target sample data into a preset network to extract visual style features, and obtain the positive sample data vector corresponding to the positive sample data, the negative sample data vector corresponding to the negative sample data, and the anchored target sample data vector.

[0120] Figure 6 This is a schematic diagram illustrating the structure of a preset network according to an exemplary embodiment. For example... Figure 6 As shown, in step S10111 above, the server can input positive sample data, negative sample data and anchor target sample data into a preset network to extract visual style features, and obtain the positive sample data vector (PosEmbedding) corresponding to the positive sample data, the negative sample data vector (NegEmbedding) corresponding to the negative sample data and the anchor target sample data pair anchor sample data vector (AnchorEmbedding).

[0121] Figure 7 This is a schematic diagram illustrating the structure of another preset network according to an exemplary embodiment. For example... Figure 7 As shown, the preset network can include at least a feature extraction layer, a feature connection layer (Concat), and a feature fusion layer. Furthermore, the positive sample data, negative sample data, and anchored target sample data each include multimodal data. Then, continuing as... Figure 5 As shown, step S10111 above may further include:

[0122] S101111. Input the multimodal data included in each of the positive sample data, negative sample data, and anchored target sample data into the feature extraction layer for visual style feature extraction processing to obtain the feature extraction results of the multimodal data included in each of the positive sample data, negative sample data, and anchored target sample data.

[0123] Optionally, the positive sample data, negative sample data, and anchored target sample data each contain their corresponding multimodal data. This multimodal data may include, but is not limited to, visual modal data, audio modal data, and text modal data.

[0124] In one specific implementation, the multimodal data contained in each sample data can be directly input into the feature extraction layer for visual style feature extraction processing to obtain the feature extraction results of the multimodal data contained in each sample data.

[0125] The following section uses visual modal data, audio modal data, and text modal data as examples to explain the process of feature extraction from the multimodal data included in each of the above-mentioned positive sample data, negative sample data, and anchored target sample data:

[0126] Continue as Figure 7 As shown, the feature extraction process for visual modality data is as follows: The embedding vector of multimodal video content represents a video using a low-dimensional vector. Through the extracted video content, the "distance" between two embeddings represents the distance between two videos, allowing for the calculation of video similarity. Video content embedding is based on the "implicit" features of the content. The video content vector contains two layers of meaning: First, representation learning, low-dimensional dense features, a one-dimensional array (e.g., a video embedding is a 128-bit floating-point data type); second, metric learning, a similarity metric vector, where the "distance" between two vectors represents the "similarity" between two objects. Continuing... Figure 7 As shown, a sequence of video frames can be input as visual modality data. The visual modality data is then processed by Temporal Segment Networks (TSNs), and image features are extracted using a backbone network (Sw inT). Next, the features extracted by the backbone network are processed using the intermediate layers of the Youtub8M-NeXtVLad network competition model to obtain image feature vectors. Finally, the image feature vectors are summed and averaged to obtain the video embedding vector, thus yielding the feature extraction result for the video modality. In the large-scale video classification task of the Youtub8M-NeXtVLad network, the model structure that aggregates frame-level features into video-level features is based on the idea of ​​decomposing high-dimensional features into a set of relatively low-dimensional vectors before aggregation. For the image-text content data of the video modality, the cover image features are directly extracted using the Transformer network. The Transformer network is a model that utilizes an attention mechanism to improve model training speed.

[0127] Continue as Figure 7As shown, the feature extraction process for audio modal data is as follows: First, the audio modal data is preprocessed to obtain a Mel spectrogram. For example, 16kHz audio from the first 10 minutes of a video is selected, and a short-time Fourier transform is performed on the audio using a 25ms Hamming window and a 10ms frame shift to obtain the spectrogram. Then, the spectrogram is mapped onto a 64th-order Mel filter bank to calculate the Mel spectrum. The Mel spectrum features are framed in 960ms intervals with no overlap between frames, each frame being 10ms long and containing 64 Mel frequency bands. After obtaining the Mel spectrogram, a feature extraction layer can be used, such as the Audio Vector Model (VGGish), to extract features from the Mel spectrogram. Because VGGish has a strong ability to express scene-related sound events, adding audio modal data significantly improves the accuracy of semantic modeling for content such as action movies and music. Finally, NextVlad is used to obtain the feature extraction results for the audio modal data.

[0128] Continue as Figure 7 As shown, the feature extraction process for text modal data involves inputting text modal data into a preset network. This input primarily includes the content title, tags, Automatic Speech Recognition (ASR), and Optical Character Recognition (OCR) results (this input is not required for image-text content). The title is the publisher's subjective description of the content, typically encompassing the high-level semantics the video aims to convey. In practical applications, many videos lack titles or their titles convey insufficient information; OCR can supplement this missing textual information. However, OCR data also presents several issues, such as inaccurate OCR recognition during scene transitions, the need for deduplication in fixed-position OCR, the retention of spoken OCR, and the deletion of news scrolling OCR. Therefore, this embodiment can perform noise reduction processing on the OCR recognition results, including filtering single-character / pure number / pure letter OCR, filtering OCR with small offsets between adjacent frame bounding boxes (bboxes) and high text repetition rates, and filtering OCR with bboxes at the bottom of the screen and small height. The use of ASR text is similar to that of OCR, employing the LICHEES model to model the text. The LICHEES model is a pre-trained language representation model (Bidirectional Encoder Representations from Transformers, BERT) based on a large-scale information flow text corpus. This model enables better extraction of semantic features from the information flow text corpus, thus yielding the feature extraction results for the text modality data.

[0129] In another specific implementation, visual style can reflect the client's emotional response to content, manifested on the client's first impression upon seeing the title and cover image. However, while the network can learn some visual style features based solely on the title and cover image, it easily confuses items that are unrelated in the classification system, resulting in lower model learning accuracy. Prior features also exhibit certain patterns in visual style; for example, content like "entertainment information" or "game highlights" inherently suggests a high probability of a lighthearted, entertaining, and youthful emotional response. Therefore, data categories (e.g., primary and / or secondary categories) can be incorporated during training. Experimental results show that adding secondary categories improves the network's prediction accuracy, while adding primary categories does not. This is because primary categories have a relatively large granularity regarding emotional response; for example, content may contain positive, negative, or subdued content, and using primary categories can easily confuse content with different emotional responses. However, secondary categories have a smaller granularity; for example, game highlights or game commentary can be aggregated into a youthful emotional response, thus secondary categories have higher accuracy in visual style prediction.

[0130] Based on this, in a feasible embodiment, in S101111 above, the input of the multimodal data included in each of the positive sample data, negative sample data, and anchored target sample data into the feature extraction layer for visual style feature extraction processing, to obtain the feature extraction results of the multimodal data included in each of the positive sample data, negative sample data, and anchored target sample data, may include:

[0131] Obtain the data categories of multimodal data included in each of the positive sample data, negative sample data, and anchored target sample data.

[0132] The multimodal data included in each of the positive sample data, negative sample data, and anchor target sample data, as well as the data categories of the multimodal data, are input into the feature extraction layer for visual style feature extraction processing, to obtain the feature extraction results of the multimodal data included in each of the positive sample data, negative sample data, and anchor target sample data.

[0133] As mentioned above, since secondary categories have high accuracy in visual style prediction, we can obtain the data categories of the multimodal data included in the positive sample data, negative sample data, and anchor target sample data, and extract the secondary category data from each data category. The secondary category data and the multimodal data of each sample data are then input into the feature extraction layer for visual style feature extraction. This yields the feature extraction results for the multimodal data included in each sample data, ensuring that the vectors extracted by the network represent features that are more focused on visual style. This allows the network to learn the differences in visual style, further improving the training accuracy of the visual style network.

[0134] S101113. Input the feature extraction results of the multimodal data into the feature connection layer for visual style feature connection processing to obtain the feature connection results of the multimodal data included in the positive sample data, negative sample data and anchored target sample data respectively.

[0135] S101115. Input the feature concatenation results of the multimodal data into the feature fusion layer for visual style feature fusion processing to obtain the feature fusion results of the multimodal data included in the positive sample data, negative sample data, and anchored target sample data respectively.

[0136] Continue as Figure 7 As shown, in steps S101113-S101115 above, the server can input the feature extraction results of the multimodal data into the feature connection layer, which performs visual style feature connection processing on the features to obtain the feature connection results of the multimodal data included in each of the positive sample data, negative sample data, and anchored target sample data. Then, the feature connection results of the multimodal data are input into the feature fusion layer for visual style feature fusion processing, which performs visual style feature fusion processing on the features to obtain the feature fusion results of the multimodal data included in each of the positive sample data, negative sample data, and anchored target sample data.

[0137] S101117. The feature fusion result of the multimodal data included in the positive sample data is determined as the positive sample data vector; the feature fusion result of the multimodal data included in the negative sample data is determined as the negative sample data vector; the feature fusion result of the multimodal data included in the anchored target sample data is determined as the anchored sample data vector.

[0138] Continue as Figure 7 As shown, after obtaining the feature fusion results of the multimodal data of each sample data, the server can determine the feature fusion results of the multimodal data included in the positive sample data as the positive sample data vector (PosEmbedding), the feature fusion results of the multimodal data included in the negative sample data as the negative sample data vector (NegEmbedding), and the feature fusion results of the multimodal data included in the anchor target sample data as the anchor sample data vector (AnchorEmbedding).

[0139] In information flow recommendation, since the existing content classification and tagging system cannot control the overall visual style, and since single-modal data cannot describe the visual style of the content, this application proposes a multimodal abstract semantic description of visual style. Feature extraction, feature connection, and feature fusion are performed on the multimodal data included in each sample data to obtain positive sample data vectors, negative sample data vectors, and anchor sample data vectors. This effectively portrays the first emotional feeling of the client when seeing the content, making it easier to identify the content from higher and multiple dimensions. The vectors extracted by the network represent features that focus more on visual style, so that the network can learn the differences in visual style, thereby further improving the training accuracy of the visual style network.

[0140] S10113. Calculate the first difference between the anchored sample data vector and the negative sample data vector, and the second difference between the anchored sample data vector and the positive sample data vector.

[0141] S10115. Calculate the loss data based on the first difference and the second difference.

[0142] Optionally, in steps S10113-S10115 above, this embodiment of the application may employ Triplet Loss to constrain the feature vector space distribution. Triplet Loss is a loss function in deep learning that calculates sample similarity by optimizing the distance between the anchor example and the positive example to be less than the distance between the anchor example and the negative example.

[0143] Continue as Figure 6 As shown, the first difference (disn) between the anchor sample data vector and the negative sample data vector, and the second difference (disp) between the anchor sample data vector and the positive sample data vector can be calculated. The Triplet Loss function is then used to calculate the loss data based on the first difference (disn) and the second difference (disp). Since Triplet Loss is a loss function in deep learning, it can improve the accuracy of loss data determination by optimizing the distance between the anchor example and the positive example to be less than the distance between the anchor example and the negative example. Using Triplet Loss to calculate the loss data can improve the accuracy of the loss data determination, thereby improving the training accuracy of the visual style network.

[0144] In a feasible embodiment, since the entire algorithm process, from constructing the corpus to training the visual style network, and then extracting content features and clustering based on the trained visual style network, consumes a certain amount of time. To accelerate the network's learning of the main visual style features and achieve rapid convergence, thereby reducing the overall algorithm time and its impact on system resources, an Online Hard Triplet strategy can be used to calculate the loss data. Accordingly, continuing as follows... Figure 5 As shown, in step S10115 above, the calculation of loss data based on the first difference and the second difference may include:

[0145] S101151. Obtain edge parameters; edge parameters are parameters that characterize whether the first distance and the second distance satisfy the third preset condition. The first distance is the distance between the anchored sample data vector and the negative sample data vector, and the second distance is the distance between the anchored sample data vector and the positive sample data vector.

[0146] S101153. Calculate the loss data based on the first difference, the second difference, and the edge parameters.

[0147] Optionally, the Triplet triple consists of an anchored sample data vector (origin), a positive sample data vector (sample points of the same class), and a negative sample data vector (sample points of different classes). This triple can satisfy the following formula in the feature space:

[0148] L=max(d(a,p)-d(a,n)+margin,0);

[0149] Here, distance L represents the loss data, d(a,p) refers to the difference between the anchored sample data vector and the positive sample data vector, d(a,n) refers to the difference between the anchored sample data vector and the negative sample data vector, and margin refers to the edge parameter, which represents the satisfaction of a third preset condition between the first distance between the anchored sample data vector and the positive sample data vector, and the second distance between the anchored sample data vector and the negative sample data vector. For example, the third preset condition can be the minimum interval between the first distance and the second distance. From the above formula, it can be seen that the data can be the maximum value between (d(a,p)-d(a,n)+margin) and 0.

[0150] In steps S101151-S101153 above, the server can obtain the edge parameter and calculate the loss data based on the first difference, the second difference, and the edge parameter according to the above formula. To minimize the distance L, d(a,p) / d(a,n) can satisfy the following relationship:

[0151] d(a, P)→0, d(a, n)>margin

[0152] Optionally, during the construction of the Triplet, many easy triplets will exist, resulting in zero loss data. These easy triplets are corpora that the network does not need in later stages. To accelerate network training efficiency and reduce the consumption of system resources during training, hard triplets (triples that are guaranteed to be misidentified) can be used in later training epochs. Using this type of corpus in later training allows the network to learn the features of hard samples more quickly, accelerating training efficiency and reducing the consumption of system resources. As an example, the specific training epochs corresponding to later stages of network training can be set according to actual business needs. For example, hard triplets can be used for network training after 10 epochs. As an example, the server can determine which triplets are guaranteed to be misidentified by comparing the results of machine recognition and human recognition.

[0153] S10117. Adjust the network parameters of the preset network according to the loss data until the preset training termination condition is met to obtain the visual style network.

[0154] Optionally, after the server obtains the loss data, it determines whether the loss data meets the preset training termination condition, or whether the number of times the network has been trained meets the preset training termination condition. If either of these conditions is met, the network training ends, and the trained network is used as the visual style network. If neither of these conditions is met, the network training process is repeated until the loss data meets the preset training termination condition, or the number of times the network has been trained meets the preset training termination condition.

[0155] The input to the network training process in this embodiment is a triplet consisting of positive sample data, negative sample data, and anchor target sample data. Since this triplet is constructed using a combination of unsupervised and weakly supervised methods, it can extract the content features of the content itself. Using this triplet as input to the preset network allows the vectors extracted by the network to represent features that are more focused on visual style. This enables the network to learn the differences in visual style, thereby improving the training accuracy of the visual style network. Simultaneously, this embodiment can employ Triplet Loss to constrain the feature vector space distribution. Since Triplet Loss is a loss function in deep learning, it can optimize the distance between anchor examples and positive examples to be less than the distance between anchor examples and negative examples. Using Triplet Loss to calculate the loss data can improve the accuracy of loss data determination, thereby improving the training accuracy of the visual style network. Furthermore, the combination of unsupervised and weakly supervised methods to construct triples enables the definition and differentiation of high-level semantic pseudo-labels for recommended content, allowing for targeted recommendations and distribution to client objects with corresponding tastes. This improves the overall consumer experience and compensates for the shortcomings of existing methods that primarily recommend and distribute content based on client object interests and themes. Moreover, it can fully utilize existing partial data and object interaction data of terminal objects without the need for data annotation, thereby improving the efficiency of network training and reducing the cost of network training.

[0156] The following describes the process of using a visual style network to identify the visual style of content in a content pool:

[0157] Figure 8 This is a flowchart illustrating a visual style recognition method according to an exemplary embodiment, such as... Figure 8 As shown, the visual style recognition method may include:

[0158] S201. Obtain at least two pieces of data to be processed.

[0159] S203. Input at least two data points to be processed into a visual style network for visual style recognition, and obtain visual style feature vectors corresponding to each of the at least two data points to be processed.

[0160] S205. Cluster the visual style feature vectors corresponding to at least two data points to be processed, and obtain the visual style feature clustering results corresponding to at least two data points to be processed.

[0161] S207. Send the visual style feature clustering results to the client so that the client can recommend information based on the visual style feature clustering results.

[0162] Optionally, at least two data points in S201 above can be content from all enabled content pools. After the server obtains the visual style network through the above training method, the server can, according to step S203 above, use the visual style network to vectorize the content from all enabled content pools, that is, extract visual style feature vectors (Embeddings), and, as described in step S205 above, cluster the extracted visual style feature vectors (Embeddings) to obtain visual style feature clustering results. Each visual style feature clustering result can be considered as a pseudo-label of a visual style, such as a lighthearted entertainment visual style cluster, a low-key visual style cluster, a negative energy visual style cluster, a positive energy cluster, etc. As described in step S207 above, the server can send the visual style feature clustering results to the client, and the client uses the visual style feature clustering results as features for content recommendation and distribution.

[0163] For example, in S205 above, various methods can be used to cluster the visual style feature vectors, and no specific limitation is made here. In one implementation, K-means clustering can be used. The principle of K-means is as follows: K-means uses minimizing the squared error between the sample and the particle as the objective function. The sum of the squared distance errors between the particles in each cluster and the sample points within the cluster is called the distortion. For a cluster, the lower the distortion, the more compact the members within the cluster; the higher the distortion, the looser the cluster structure. The distortion decreases as the number of categories increases, but for data with a certain degree of discriminative power, the distortion will be greatly improved when a certain critical point is reached, and then slowly decrease. This critical point can be considered as the point with better clustering performance. The optimal number of clusters can be determined by referring to the K-means elbow rule. The principle of the K-means elbow rule is as follows: Assuming K is the number of clusters, SSE is the sum of the distances from all points to the assigned class center. The closer the clustering algorithm fits the true distribution, the smaller the SSE. The elbow rule states that when the selected cluster size K is less than the actual cluster size, increasing K will significantly decrease the SSE. However, when K exceeds the actual cluster size, further increases in K will not result in a more pronounced decrease in SSE. The actual K will show an inflection point on the SSE trend chart. Figure 9 This is a schematic diagram illustrating the principle of the K-means elbow rule according to an exemplary embodiment. Figure 9 As shown, through clustering experiments, with a cluster size span of 5, a slightly noticeable inflection point was observed around 100 clusters. Ultimately, 100 clusters were selected for K-means clustering. The above method can be used to determine clustering parameters under different distribution conditions.

[0164] Figure 10This is a schematic diagram illustrating a visual style feature clustering result according to an exemplary embodiment. These visual style feature clustering results can be used for client-side content recommendation and distribution, such as... Figure 10 As shown, the results of this visual style feature clustering can include: high-key and eye-catching style cluster, cute pet visual style cluster, cute baby visual style cluster, serious and formal content cluster, negative content cluster, consistent writing style cluster, and small knowledge and tips cluster, etc.

[0165] In this embodiment, a trained visual style network is used to vectorize all content in the enabled content pool, and the extracted visual style feature embeddings are clustered to obtain visual style feature clustering results. The extraction process of these visual style feature clustering results strips away the content features themselves, and can be considered as a set of pseudo-tags to be used during recommendation. The server sends this set of pseudo-tags to the client, which uses it as the basis for content recommendation and distribution. Since the visual style feature clustering results can label all styles present in the content data, it can support some styles that actually exist but have not yet been included in the supervised tag set, thus covering a more comprehensive tag set. Sending the style feature clustering results to the client allows the client to recommend information based on the visual style feature clustering results, thereby improving the recommendation scope, recommendation accuracy, and recommendation effect, and reducing the consumption of system resources in the recommendation process.

[0166] Figure 11 This is a schematic diagram illustrating a system for visual style recognition based on a combination of unsupervised and weakly supervised visual style network training, according to an exemplary embodiment. Figure 11 As shown, the system may include:

[0167] I. Content Production and Consumption

[0168] (1) PGC or UGC, MCN or professional user-generated content (PUGC) content producers provide local or photographed text and image content, video or image collection content through mobile terminal or backend interface application program interface (API) system. These are the main sources of content for distribution.

[0169] (2) By communicating with the upstream and downstream content interface services, the upload server interface address is obtained first, and then the local file is uploaded. During the shooting process, the local text and image content can be matched with music, filter templates and text and image beautification functions, etc.

[0170] (3) As a consumer, communicate with the upstream and downstream content interface server to obtain the index information of the accessed text or video file (usually the address of the accessed content), and then download the corresponding streaming media file and play it through the local player;

[0171] (4) At the same time, the interactive data, stuttering, loading time, playback clicks, etc. of the user during the upload and download process are reported to the server;

[0172] (5) Interactive information of consumer content, focusing on interactive information such as playback, click, browsing, swiping, liking, forwarding, collecting, and sharing of content, which are reported through interactive data and distribution statistics reporting interface.

[0173] II. Uplink and Downlink Content Interface Server

[0174] (1) Communicate directly with the content production end. The content submitted from the front end, usually the title, publisher, summary, cover image, publication time, or photos and text, is directly sent to the server and stored in the content database.

[0175] (2) Submit the uploaded file to the dispatch center server for subsequent content processing and transfer;

[0176] III. Content Database

[0177] (1) The core database of content, where all the metadata of the content published by producers is stored. The focus is on the metadata of the content itself, such as file size, cover image link, bitrate, file format, title, publication time, author, video file size, video format, whether it is original or first publication, as well as the classification of content during the manual review process (including first, second and third level classification and tag information, such as a video explaining mobile phones, the first level classification is technology, the second level classification is smartphones, and the third level classification is domestic mobile phones);

[0178] (2) When the uplink and downlink content interface service receives the video file, it performs standard transcoding interaction on the content. After the transcoding is completed, it asynchronously returns metadata, mainly including file size, bit rate, specifications, cover image capture, frame extraction, etc. This information will be stored in the content database.

[0179] (3) During the manual review process, information in the content database will be read, and the results and status of the manual review will also be sent back to the content database to update the meta information of the content in the content database.

[0180] (4) The scheduling center mainly handles content processing through machine processing and manual review. The core of machine processing is to call the deduplication service to check for completely duplicate and similar content. The deduplication results are written to the content database. Content processed by the machine will not be subject to manual reprocessing.

[0181] IV. Dispatch Center

[0182] (1) Responsible for the entire scheduling process of content flow, receiving content entering the content database through the upstream and downstream content interface servers, and then obtaining metadata from the internal database;

[0183] (2) Schedule the manual review system and the machine processing system, and control the scheduling order and priority;

[0184] (3) For video content, the technicians responsible for video deduplication services can process the video content as follows: process different bitrates, different resolutions, different sizes, partial black screens, whether there are filters, insert some advertising content in similar video content, and crop the beginning and end of the video, so as to effectively build the content that needs to be processed in the manual review process.

[0185] (4) Schedule video content that needs to be manually reviewed into the manual review system and filter out duplicate content;

[0186] (5) Content that has passed the manual review system is enabled and then directly provided to the content consumers on the client side through the content export distribution service (usually a recommendation engine, search engine or operation), that is, the consumer side obtains the content index information;

[0187] (6) Call the visual style recognition service, directly assign the corresponding pseudo-label to the type of visual style, and then the label of the same type of visual style can be used as a content aggregation recommendation of visual style tone, which can better attract users and promote the cold start and spread of type of content, and improve the experience; for example, in the above steps S203-S205, visual style recognition can be performed through the visual style network to obtain the visual style feature clustering results.

[0188] V. Manual Review System

[0189] (1) It is necessary to read the original information of the text and image content itself in the content database. It is usually a complex system developed based on the World Wide Web (WEB) database. The system uses manual methods to perform a preliminary filtering of whether the text and image content involves non-compliant characteristics.

[0190] (2) On the basis of the initial review, the content is reviewed a second time, mainly to classify and label or confirm the content. Since the video content itself is entirely processed by machine learning, but machine learning is not yet fully mature, it is necessary to conduct a second manual review on the basis of machine processing. Through human-machine collaboration, the accuracy and efficiency of the video content labeling, especially the tags, can be improved.

[0191] VI. Deduplication Service

[0192] (1) Provide deduplication services for text, images and videos. The main function is to vectorize the text, images and videos, then create an index for the vectors, and then determine the degree of repetition by comparing the distance between the vectors.

[0193] (2) For text and image content, the text and image are usually vectorized (BERT vectorization). Before all deduplication tasks, the title short text is deduplicated first to reduce the amount of computation in subsequent processing.

[0194] VII. Content Storage Service

[0195] (1) Usually, it is a group of storage servers that are widely distributed and close to the client object. There are usually CDN acceleration servers on the periphery of the content storage service for distributed caching acceleration. The video content uploaded by the content producer is saved through the upstream and downstream content interface servers, and the original cover image of the graphic content is also saved.

[0196] (2) After obtaining the content index information, client consumers can also directly access the video content storage server to download the corresponding content;

[0197] (3) In addition to serving as a data source for external services, it also serves as a data source for internal services, allowing the download file system to obtain raw video data for related processing. The paths for internal and external data sources are usually deployed separately to avoid mutual interference. For example, the sample data in step S101 and the data to be processed in step S201 can both be obtained from this content storage service.

[0198] 8. Weakly supervised and unsupervised data cleaning services

[0199] (1) Following the different methods of combining unsupervised and weakly supervised data mining described above, construct corpus information by combining user post-interaction data, and simultaneously perform corresponding filtering and cleaning processes. For specific procedures, please refer to steps S103-S107 above, which will not be repeated here.

[0200] 9. Visual Style Network

[0201] (1) Following the unsupervised model described above, a visual style feature space is constructed in the visual style dimension of content consumption. The content features themselves are stripped away, and the visual style vector Em bedding of the content is extracted as an intermediate product for downstream content use. For the specific process, please refer to steps S201-S205 above, which will not be repeated here.

[0202] 10. Visual Style Clustering Model

[0203] (1) Obtain the visual style network for the content visual style, and vectorize all enabled content pool content, i.e., extract the style feature vector Embedding, and perform K-means clustering. For the specific process, please refer to step S207 above, which will not be repeated here.

[0204] XI. Visual Style Recognition Service

[0205] (1) The above content visual style clustering model is serviced and communicated with the scheduling center service to call the corresponding visual style recognition service to process the content and existing content on the processing link.

[0206] (2) The processing result serves as a visual style feature marker for the content. Based on this marker, content pools with the same style theme can be recommended and distributed accordingly, with corresponding dispersion and tone control, thereby improving the recommendation effect. For details, please refer to step S207 above, which will not be repeated here.

[0207] Figure 12 This is a block diagram illustrating a visual style network training apparatus according to an exemplary embodiment. Figure 12 As shown, the device may include at least:

[0208] The sample data pair and interactive data acquisition module 301 is used to acquire at least two sample data pairs and the object interactive data corresponding to each data in each sample data pair; the display data of each data in each sample data pair is the same, and the content of each data in each sample data pair matches.

[0209] The initial sample data pair determination module 303 is used to determine an initial sample data pair from at least two sample data pairs based on object interaction data; the initial sample data pair includes an initial positive sample data pair and an initial negative sample data pair; the object interaction data corresponding to each data in the initial positive sample data pair satisfies a first preset condition; the object interaction data corresponding to each data in the initial negative sample data pair satisfies a second preset condition.

[0210] The target candidate object acquisition module 305 is used to display the data contained in the initial sample data pair to the candidate object set, and to acquire the target candidate objects that interact with the data contained in the initial sample data pair from the candidate object set.

[0211] The target sample data pair determination module 307 is used to determine target sample data pairs from the initial sample data pairs based on the object attribute information of the target candidate objects. The target sample data pairs include target positive sample data pairs determined from the initial positive sample data pairs and target negative sample data pairs determined from the initial negative sample data pairs. The visual styles of the data in the target positive sample data pairs match, while the visual styles of the data in the target negative sample data pairs do not match.

[0212] Module 309 is used to construct positive sample data, negative sample data, and anchored target sample data based on the target sample data pair; the content of the positive sample data matches the content of the anchored target sample data, while the content of the negative sample data does not match the content of the anchored target sample data.

[0213] Training module 3011 is used to calculate loss data based on the difference between positive sample data and anchored target sample data, and the difference between negative sample data and anchored target sample data, and to train a preset network based on the loss data to obtain a visual style network.

[0214] In an optional embodiment, the initial sample data pair determination module 303 may include:

[0215] The initial positive and negative sample pair determination unit is used to determine an initial positive sample data pair and an initial negative sample data pair from at least two sample data pairs based on object interaction data; the object interaction data corresponding to each number contained in the initial positive sample data pair are matched; the difference between the object interaction data corresponding to each data contained in the initial negative sample data pair is greater than a preset difference threshold.

[0216] The initial sample data pair generation unit is used to determine the initial positive sample data pair and the initial negative sample data pair, which are the initial sample data pairs.

[0217] In an optional embodiment, the number of initial sample data pairs is at least two, each initial sample data pair includes first data and second data, the target candidate objects include a first target candidate object that interacts with the first data and a second target candidate object that interacts with the second data, and the target sample data pair determination module 307 includes:

[0218] An attribute information acquisition unit is used to acquire first object attribute information of a first target candidate object and second object attribute information of a second target candidate object.

[0219] The intersection-union ratio determination unit is used to determine the intersection-union ratio of object attribute information between the first object attribute information and the second object attribute information;

[0220] The target positive and negative sample data pair determination unit is used to determine the initial positive sample data pairs whose object attribute information intersection-union ratio is greater than the preset intersection-union ratio threshold as target positive sample data pairs, and to determine the initial negative sample data pairs whose object attribute information intersection-union ratio is less than or equal to the preset intersection-union ratio threshold as target negative sample data pairs.

[0221] The target sample data pair generation unit is used to determine the target positive sample data pair and the target negative sample data pair, which are the target sample data pairs.

[0222] In an optional embodiment, the construction module 309 includes:

[0223] Anchored target sample data determination unit, used to determine any one of the data in the target positive sample data pair as anchored target sample data;

[0224] The positive sample data determination unit is used to identify data whose content matches the content of the anchored target sample data in the target positive sample data pair as positive sample data.

[0225] The negative sample data determination unit is used to identify data in the target negative sample data pair whose content does not match the content of the anchored target sample data as negative sample data.

[0226] In an optional embodiment, the training module 3011 includes:

[0227] The sample data vector generation submodule is used to input positive sample data, negative sample data and anchored target sample data into a preset network for visual style feature extraction, and obtain the positive sample data vector corresponding to the positive sample data, the negative sample data vector corresponding to the negative sample data and the anchored target sample data vector to the anchored sample data.

[0228] The difference calculation submodule is used to calculate the first difference between the anchor sample data vector and the negative sample data vector, and the second difference between the anchor sample data vector and the positive sample data vector.

[0229] The loss data calculation submodule is used to calculate the loss data based on the first difference and the second difference.

[0230] The visual style network generation submodule is used to adjust the network parameters of the preset network based on the loss data until the preset training termination condition is met, thus obtaining the visual style network.

[0231] In an optional embodiment, the preset network includes a feature extraction layer, a feature connection layer, and a feature fusion layer, wherein the positive sample data, negative sample data, and anchored target sample data each include multimodal data;

[0232] The sample data vector generation submodule includes:

[0233] The feature extraction unit is used to input the multimodal data included in the positive sample data, negative sample data and anchor target sample data into the feature extraction layer for visual style feature extraction processing, and obtain the feature extraction results of the multimodal data included in the positive sample data, negative sample data and anchor target sample data respectively;

[0234] The feature connection unit is used to input the feature extraction results of multimodal data into the feature connection layer for visual style feature connection processing, and obtain the feature connection results of the multimodal data included in the positive sample data, negative sample data and anchored target sample data respectively;

[0235] The feature fusion unit is used to input the feature concatenation results of multimodal data into the feature fusion layer for visual style feature fusion processing, and obtain the feature fusion results of the multimodal data included in the positive sample data, negative sample data and anchored target sample data respectively;

[0236] The vector determination unit is used to determine the feature fusion result of the multimodal data included in the positive sample data as the positive sample data vector; to determine the feature fusion result of the multimodal data included in the negative sample data as the negative sample data vector; and to determine the feature fusion result of the multimodal data included in the anchor target sample data as the anchor sample data vector.

[0237] In an optional embodiment, the feature extraction unit includes:

[0238] The category acquisition sub-unit is used to acquire the data categories of the multimodal data included in each of the positive sample data, negative sample data, and anchored target sample data;

[0239] The category data input subunit is used to input the multimodal data included in each of the positive sample data, negative sample data, and anchor target sample data, as well as the data categories of the multimodal data, into the feature extraction layer for visual style feature extraction processing, and obtain the feature extraction results of the multimodal data included in each of the positive sample data, negative sample data, and anchor target sample data.

[0240] In an optional embodiment, the loss data calculation submodule includes:

[0241] An edge parameter acquisition unit is used to acquire edge parameters; the edge parameters are parameters that characterize whether the first distance and the second distance satisfy a third preset condition, the first distance is the distance between the anchored sample data vector and the negative sample data vector, and the second distance is the distance between the anchored sample data vector and the positive sample data vector;

[0242] The loss data calculation unit is used to calculate loss data based on the first difference, the second difference, and the edge parameters.

[0243] Figure 13 This is a block diagram illustrating a visual style recognition device according to an exemplary embodiment. Figure 13 As shown, the device may include at least:

[0244] The data acquisition module 401 is used to acquire at least two data items to be processed.

[0245] The recognition module 403 is used to input at least two data to be processed into the visual style network for visual style recognition, and obtain the visual style feature vectors corresponding to each of the at least two data to be processed.

[0246] Clustering module 405 is used to cluster the visual style feature vectors corresponding to at least two data to be processed, and to obtain the visual style feature clustering results corresponding to at least two data to be processed.

[0247] The sending module 407 is used to send the visual style feature clustering results to the client so that the client can make information recommendations based on the visual style feature clustering results.

[0248] It should be noted that the device embodiments provided in this application are based on the same inventive concept as the method embodiments described above.

[0249] This application also provides an electronic device for visual style network training or visual style recognition. The electronic device includes a processor and a memory. The memory stores at least one instruction or at least one program. The processor loads and executes the at least one instruction or at least one program to implement the visual style network training method or visual style recognition method provided in any of the above embodiments.

[0250] Embodiments of this application also provide a computer-readable storage medium that can be disposed in a terminal to store at least one instruction or at least one program for implementing an object processing method in the method embodiments. The at least one instruction or at least one program is loaded and executed by a processor to implement the visual style network training method or visual style recognition method provided in the above method embodiments.

[0251] Optionally, in the embodiments of this specification, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0252] The memory described in this specification can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for functions, etc.; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0253] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the visual style network training method or visual style recognition method provided in the above-described method embodiments.

[0254] The visual style network training method or visual style recognition method embodiment provided in this application can be executed on a terminal, computer terminal, server, or similar computing device. Taking running on a server as an example, Figure 14 This is a hardware structure block diagram of a server for visual style network training or visual style recognition, provided according to an exemplary embodiment. For example... Figure 14As shown, the server 500 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 510 (CPUs 510 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 530 for storing data, and one or more storage media 520 (e.g., one or more mass storage devices) for storing application programs 523 or data 522. The memory 530 and storage media 520 may be temporary or persistent storage. The program stored in the storage media 520 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 510 may be configured to communicate with the storage media 520 and execute the series of instruction operations stored in the storage media 520 on the server 500. Server 500 may also include one or more power supplies 560, one or more wired or wireless network interfaces 550, one or more input / output interfaces 540, and / or one or more operating systems 521, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0255] The input / output interface 540 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 500. In one example, the input / output interface 540 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 540 may be a radio frequency (RF) module for wireless communication with the Internet.

[0256] Those skilled in the art will understand that Figure 14 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 500 may also include... Figure 14 The more or fewer components shown, or having the same Figure 14 The different configurations shown.

[0257] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0258] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and server embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0259] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0260] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A visual style network training method, characterized in that, The method comprises: obtaining at least two sample data pairs and object interaction data corresponding to each data included in each sample data pair; each data included in each sample data pair has the same display data, and the content of each data included in each sample data pair is matched; determining an initial sample data pair from the at least two sample data pairs according to the object interaction data; the initial sample data pair comprises an initial positive sample data pair and an initial negative sample data pair; the object interaction data corresponding to each data included in the initial positive sample data pair satisfies a first preset condition; the object interaction data corresponding to each data included in the initial negative sample data pair satisfies a second preset condition; displaying each data included in the initial sample data pair to a candidate object set, and obtaining a target candidate object that interacts with each data included in the initial sample data pair from the candidate object set; determining a target sample data pair from the initial sample data pair according to object attribute information of the target candidate object; the target sample data pair comprises a target positive sample data pair determined from the initial positive sample data pair, and a target negative sample data pair determined from the initial negative sample data pair; the visual style of each data included in the target positive sample data pair is matched; the visual style of each data included in the target negative sample data pair is not matched; constructing positive sample data, negative sample data and anchor target sample data according to the target sample data pair; the content of the positive sample data is matched with the content of the anchor target sample data; the content of the negative sample data is not matched with the content of the anchor target sample data; inputting the positive sample data, the negative sample data and the anchor target sample data into a preset network to extract visual style features, to obtain a positive sample data vector corresponding to the positive sample data, a negative sample data vector corresponding to the negative sample data, and an anchor sample data vector corresponding to the anchor target sample data; calculating a first difference between the anchor sample data vector and the negative sample data vector, and a second difference between the anchor sample data vector and the positive sample data vector; calculating loss data according to the first difference and the second difference; adjusting network parameters of the preset network according to the loss data until a preset training end condition is met, to obtain a visual style network.

2. The method of claim 1, wherein, The method comprises: determining the initial positive sample data pair and the initial negative sample data pair from the at least two sample data pairs based on the object interaction data; the object interaction data corresponding to each data included in the initial positive sample data pair is matched; the difference between the object interaction data corresponding to each data included in the initial negative sample data pair is greater than a preset difference threshold; determining the initial positive sample data pair and the initial negative sample data pair as the initial sample data pair.

3. The method of claim 1, wherein, The number of the initial sample data pairs is at least two, each of the initial sample data pairs comprises first data and second data, the target candidate object comprises a first target candidate object interacting with the first data and a second target candidate object interacting with the second data, and the target sample data pair is determined from the initial sample data pairs according to object attribute information of the target candidate object, comprising: obtaining first object attribute information of the first target candidate object and second object attribute information of the second target candidate object; determining an object attribute information intersection-over-union between the first object attribute information and the second object attribute information to obtain an object attribute information intersection-over-union of each of the initial sample data pairs; determining an initial positive sample data pair with an object attribute information intersection-over-union greater than a preset intersection-over-union threshold as the target positive sample data pair, and determining an initial negative sample data pair with an object attribute information intersection-over-union less than or equal to the preset intersection-over-union threshold as the target negative sample data pair; determining the target positive sample data pair and the target negative sample data pair as the target sample data pair.

4. The method of claim 1, wherein, The target sample data pair is used to construct positive sample data, negative sample data and anchor target sample data, comprising: determining any one of the target positive sample data pair as the anchor target sample data; determining data in the target positive sample data pair with content matching that of the anchor target sample data as the positive sample data; determining data in the target negative sample data pair with content not matching that of the anchor target sample data as the negative sample data.

5. The method of claim 1, wherein, The preset network comprises a feature extraction layer, a feature connection layer and a feature fusion layer, and the positive sample data, the negative sample data and the anchor target sample data each comprise multi-modal data; The positive sample data, the negative sample data and the anchor target sample data are input into a preset network for visual style feature extraction to obtain positive sample data vectors corresponding to the positive sample data, negative sample data vectors corresponding to the negative sample data and anchor sample data vectors corresponding to the anchor target sample data, comprising: the multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data are input into the feature extraction layer for visual style feature extraction processing to obtain feature extraction results of the multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data; the feature extraction results of the multi-modal data are input into the feature connection layer for visual style feature connection processing to obtain feature connection results of the multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data; the feature connection results of the multi-modal data are input into the feature fusion layer for visual style feature fusion processing to obtain feature fusion results of the multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data; The feature fusion result of the multi-modal data included in the positive sample data is determined as the positive sample data vector; the feature fusion result of the multi-modal data included in the negative sample data is determined as the negative sample data vector; and the feature fusion result of the multi-modal data included in the anchor target sample data is determined as the anchor sample data vector.

6. The method of claim 5, wherein, The multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data are input into the feature extraction layer for visual style feature extraction processing to obtain the feature extraction result of the multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data, including: The data categories of the multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data are obtained; The multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data, and the data categories of the multi-modal data are input into the feature extraction layer for visual style feature extraction processing to obtain the feature extraction result of the multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data.

7. The method of claim 1, wherein, The loss data is calculated according to the first difference and the second difference, including: An edge parameter is obtained; the edge parameter is a parameter representing that the first distance and the second distance satisfy a third preset condition, the first distance is the distance between the anchor sample data vector and the negative sample data vector, and the second distance is the distance between the anchor sample data vector and the positive sample data vector; The loss data is calculated according to the first difference, the second difference and the edge parameter.

8. A visual style recognition method, characterized by, The method includes: Obtaining at least two to-be-processed data; Inputting the at least two to-be-processed data into a visual style network for visual style recognition to obtain a visual style feature vector corresponding to each of the at least two to-be-processed data; Clustering the visual style feature vectors corresponding to the at least two to-be-processed data to obtain a visual style feature clustering result corresponding to the at least two to-be-processed data; Sending the visual style feature clustering result to a client to enable the client to perform information recommendation based on the visual style feature clustering result; The visual style network is trained by using the visual style network training method in any one of claims 1 to 7.

9. An apparatus for training a visual style network, comprising: The device includes: A sample data pair and interaction data acquisition module is configured to acquire at least two sample data pairs and object interaction data corresponding to each data included in each sample data pair; the display data of each data included in each sample data pair is the same, and the content of each data included in each sample data pair is matched. The initial sample data pair determination module is configured to determine an initial sample data pair from the at least two sample data pairs according to the object interaction data; the initial sample data pair includes an initial positive sample data pair and an initial negative sample data pair; the object interaction data corresponding to each data included in the initial positive sample data pair satisfies a first preset condition; and the object interaction data corresponding to each data included in the initial negative sample data pair satisfies a second preset condition. The target candidate object acquisition module is configured to display each data included in the initial sample data pair to a candidate object set, and acquire a target candidate object that interacts with each data included in the initial sample data pair from the candidate object set. The target sample data pair determination module is configured to determine a target sample data pair from the initial sample data pair according to object attribute information of the target candidate object; the target sample data pair includes a target positive sample data pair determined from the initial positive sample data pair, and a target negative sample data pair determined from the initial negative sample data pair; the visual style of each data included in the target positive sample data pair is matched; and the visual style of each data included in the target negative sample data pair is not matched. The construction module is configured to construct a positive sample data, a negative sample data and an anchor target sample data according to the target sample data pair; the content of the positive sample data is matched with the content of the anchor target sample data; and the content of the negative sample data is not matched with the content of the anchor target sample data. The training module is configured to calculate a loss data based on a difference between the positive sample data and the anchor target sample data, and a difference between the negative sample data and the anchor target sample data, train a preset network based on the loss data, and obtain a visual style network; the training module includes: a sample data vector generation submodule configured to input the positive sample data, the negative sample data and the anchor target sample data into a preset network for visual style feature extraction, to obtain a positive sample data vector corresponding to the positive sample data, a negative sample data vector corresponding to the negative sample data, and an anchor sample data vector corresponding to the anchor target sample data; a difference calculation submodule configured to calculate a first difference between the anchor sample data vector and the negative sample data vector, and a second difference between the anchor sample data vector and the positive sample data vector; a loss data calculation submodule configured to calculate a loss data according to the first difference and the second difference; and a visual style network generation submodule configured to adjust network parameters of the preset network according to the loss data until a preset training end condition is met, to obtain a visual style network.

10. The apparatus of claim 9, wherein, The initial sample data pair determination module includes: The initial positive and negative sample pair determination unit is configured to determine the initial positive sample data pair and the initial negative sample data pair from the at least two sample data pairs based on the object interaction data; the object interaction data corresponding to each data included in the initial positive sample data pair matches; and a difference between the object interaction data corresponding to each data included in the initial negative sample data pair is greater than a preset difference threshold. The initial sample data pair generation unit is configured to determine the initial positive sample data pair and the initial negative sample data pair as the initial sample data pair.

11. The apparatus of claim 9, wherein, The number of the initial sample data pairs is at least two, each of the initial sample data pairs includes first data and second data, the target candidate object includes a first target candidate object interacting with the first data and a second target candidate object interacting with the second data, and the target sample data pair determination module includes: The attribute information acquisition unit is configured to acquire first object attribute information of the first target candidate object and second object attribute information of the second target candidate object. The intersection-over-union ratio determination unit is configured to determine an object attribute information intersection-over-union ratio between the first object attribute information and the second object attribute information to obtain an object attribute information intersection-over-union ratio of each of the initial sample data pairs. The target positive and negative sample data pair determination unit is configured to determine an initial positive sample data pair with an object attribute information intersection-over-union ratio greater than a preset intersection-over-union ratio threshold as the target positive sample data pair, and determine an initial negative sample data pair with an object attribute information intersection-over-union ratio less than or equal to the preset intersection-over-union ratio threshold as the target negative sample data pair. The target sample data pair generation unit is configured to determine the target positive sample data pair and the target negative sample data pair as the target sample data pair.

12. The apparatus of claim 9, wherein, The construction module includes: The anchor target sample data determination unit is configured to determine any one of the target positive sample data pairs as the anchor target sample data. The positive sample data determination unit is configured to determine data in the target positive sample data pair with content matching that of the anchor target sample data as the positive sample data. The negative sample data determination unit is configured to determine data in the target negative sample data pair with content not matching that of the anchor target sample data as the negative sample data.

13. The apparatus of claim 9, wherein, The preset network includes a feature extraction layer, a feature connection layer, and a feature fusion layer, and the positive sample data, the negative sample data, and the anchor target sample data each include multi-modal data. The sample data vector generation submodule includes: The feature extraction unit is configured to input the multi-modal data included in the positive sample data, the negative sample data, and the anchor target sample data into the feature extraction layer for visual style feature extraction processing to obtain feature extraction results of the multi-modal data included in the positive sample data, the negative sample data, and the anchor target sample data. The feature connection unit is configured to input the feature extraction results of the multi-modal data included in the positive sample data, the negative sample data, and the anchor target sample data into the feature connection layer for feature connection processing to obtain feature connection results of the multi-modal data included in the positive sample data, the negative sample data, and the anchor target sample data. The feature fusion unit is configured to input the feature connection results of the multi-modal data included in the positive sample data, the negative sample data, and the anchor target sample data into the feature fusion layer for feature fusion processing to obtain a feature fusion result of the multi-modal data included in the positive sample data, the negative sample data, and the anchor target sample data. The sample data vector generation unit is configured to input the feature fusion result of the multi-modal data included in the positive sample data, the negative sample data, and the anchor target sample data into the sample data vector generation unit to obtain a sample data vector of the positive sample data, a sample data vector of the negative sample data, and a sample data vector of the anchor target sample data. The feature connection unit is configured to input the feature extraction result of the multi-modal data into the feature connection layer for visual style feature connection processing, so as to obtain the feature connection result of the multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data. The feature fusion unit is configured to input the feature connection result of the multi-modal data into the feature fusion layer for visual style feature fusion processing, so as to obtain the feature fusion result of the multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data. The vector determination unit is configured to determine the feature fusion result of the multi-modal data included in the positive sample data as the positive sample data vector, determine the feature fusion result of the multi-modal data included in the negative sample data as the negative sample data vector, and determine the feature fusion result of the multi-modal data included in the anchor target sample data as the anchor sample data vector.

14. The apparatus of claim 13, wherein, The feature extraction unit comprises: The category acquisition subunit is configured to acquire the data category of the multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data. The category data input subunit is configured to input the multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data, and the data category of the multi-modal data, into the feature extraction layer for visual style feature extraction processing, so as to obtain the feature extraction result of the multi-modal data included in the positive sample data, the negative sample data and the anchor target sample data.

15. The apparatus of claim 9, wherein, The loss data calculation sub-module comprises: The edge parameter acquisition unit is configured to acquire an edge parameter; the edge parameter is a parameter representing that the first distance and the second distance satisfy a third preset condition, the first distance is the distance between the anchor sample data vector and the negative sample data vector, and the second distance is the distance between the anchor sample data vector and the positive sample data vector. The loss data calculation unit is configured to calculate the loss data according to the first difference, the second difference and the edge parameter.

16. A visual style recognition apparatus, comprising: The device comprises: The to-be-processed data acquisition module is configured to acquire at least two to-be-processed data. The recognition module is configured to input the at least two to-be-processed data into a visual style network for visual style recognition, so as to obtain a visual style feature vector corresponding to each of the at least two to-be-processed data. The clustering module is configured to cluster the visual style feature vectors corresponding to the at least two to-be-processed data, so as to obtain a visual style feature clustering result corresponding to the at least two to-be-processed data. The sending module is configured to send the visual style feature clustering result to a client, so that the client performs information recommendation based on the visual style feature clustering result; wherein the visual style network is trained by using the visual style network training method in any one of claims 1 to 7.

17. An electronic device for visual style network training or visual style recognition, comprising: The electronic device comprises a processor and a memory, and the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the visual style network training method according to any one of claims 1 to 7 or the visual style recognition method according to claim 8.

Images