A recommended video collection reinforcement learning method based on attraction and repulsion mechanisms

Abstract

The application provides a recommended video collection reinforcement learning method based on a gravity and repulsion mechanism, and belongs to the technical field of data processing, and specifically comprises the following steps: a theme module presets a collection theme and related keyword labels of an intelligent agent; an input module acquires video metadata and converts the video metadata into an initial state vector; a contrast learning module maps multi-modal content to a high-dimensional space to generate a theme vector; a state module forms a feature vector describing a current environment state according to features of a collected video; an action module defines an interactive action space of an agent on a platform; a Dueling DQN module estimates expected Q values of each action in the action space under the current state according to the state feature vector; an output module selects an optimal interactive action in a behavior set according to the expected Q values to perform collection, and calculates a reward value based on a multi-dimensional reward function, and then updates network parameters. Through the scheme of the application, high-coverage collection of recommended video data is realized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a reinforcement learning method for recommending video acquisition based on the mechanism of attraction and repulsion. Background Technology

[0002] Currently, academia is attempting to optimize acquisition strategies by combining machine learning. For example, supervised learning-based classification models can identify video categories through labeled data, but they rely on a large number of labeled samples and are difficult to adapt to dynamic label changes. Collaborative filtering methods can uncover user interest associations, but they cannot directly guide acquisition decisions. Furthermore, existing research often focuses on single-dimensional optimization, lacking joint modeling of multimodal features, making it difficult to balance the diversity and coverage of acquisition strategies. In summary, facing the dynamic video ecosystem driven by recommendation algorithms, traditional methods have significant bottlenecks in terms of adaptability, semantic understanding, and coverage.

[0003] It is evident that there is an urgent need for a reinforcement learning method for recommendation video acquisition based on the attraction and repulsion mechanism, which can achieve high coverage of recommendation video data. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a recommendation video acquisition reinforcement learning method based on the attraction and repulsion mechanism, which at least partially solves the problems of poor adaptability, semantic understanding and coverage in the prior art.

[0005] In a first aspect, embodiments of the present invention provide a reinforcement learning method for recommending video acquisition based on a gravity and repulsion mechanism. The method is applied to a reinforcement learning-based recommending video acquisition system, which includes a topic module, an input module, a contrastive learning module, a state module, an action module, a Dueling DQN module, and an output module, comprising: Step 1: The topic module presets the collection topics and related keyword tags of the intelligent agent as the target guide for subsequent collection. Step 2: Based on goal orientation, the input module obtains video metadata from the video platform and converts it into an initial state vector; Step 3: The contrastive learning module performs unsupervised semantic modeling on the initial state vector, mapping the multimodal content to a high-dimensional space to generate topic vectors for representing video semantics. Step 4: The status module dynamically integrates the features of the acquired video with the topic vector to form a feature vector describing the current environmental status. Step 5: The action module defines the space of all interactive actions of the Agent on the platform, as an optional set of behaviors; Step 6: The Dueling DQN module estimates the expected Q-values ​​of each action in the action space to be performed in the current state based on the state feature vector. Step 7: The output module selects the optimal interactive action from the behavior set based on the expected Q value and performs the collection. It then calculates the reward value based on the multi-dimensional reward function of the attraction and repulsion mechanism and updates the network parameters. The reward value is used to guide the Q value estimation in subsequent steps.

[0006] According to a specific implementation of an embodiment of the present invention, the multi-dimensional reward function is used to evaluate the effect of taking action a in state s, and its expression is: ; in, , , These are the weighted coefficients for category matching, semantic discrepancy, and feature matching, respectively. For category matching degree, For semantic gap, This represents the feature matching degree.

[0007] According to a specific implementation of an embodiment of the present invention, the category matching degree The expression is: ; in, This represents the k-th category label of the current video content. Indicates category The number of times it appears in category list B. Represents a list of categories Length, Represents a list of categories The number of categories.

[0008] According to a specific implementation of an embodiment of the present invention, step 3 specifically includes: Step 3.1: The encoder fuses the visual, textual, and interactive data of the video into a unified semantic vector to capture multimodal features; Step 3.2: Through comparative learning, similar videos are clustered in the feature space, while dissimilar videos are separated. Step 3.3: Input the generated topic vector into the reward function to dynamically calculate the semantic gap V, wherein the expression for the semantic gap V is: ; in, β is the gravitational weight, β is the repulsive weight, G is the attractive force, and P is the repulsive force.

[0009] According to a specific implementation of the present invention, the calculation expressions for the attractive force G and the repulsive force P are as follows: ; ; in, The feature vector of the input video Nearest Agent Feature Center The Euclidean distance between them For scale parameters; The repulsive force intensity coefficient, This is the smoothing constant.

[0010] According to a specific implementation of an embodiment of the present invention, the Dueling DQN module decomposes the Q-value into a state value function and a dominance function, and the expression for calculating the Q-value is as follows: ; in, This indicates that action a is taken in state s at time step t. value, Represents the state value function, Represents the dominance function. It indicates the size of the action space.

[0011] According to a specific implementation of an embodiment of the present invention, the output module adopts... Strategies for choosing actions.

[0012] According to a specific implementation of an embodiment of the present invention, the method further includes a continuous learning step performed by the buffer module: A replay buffer is set up and the target network is updated periodically. The replay buffer is used to store empirical tuples. The parameters of the target network are periodically copied from the online network to stabilize the training process.

[0013] According to a specific implementation of an embodiment of the present invention, the state feature vector consists of the following four components: the matching rate between the video title and the preset keyword tags, the matching degree between the video category tags and the collected topic, the normalized interaction rate index, and the category after the video duration is binned and encoded.

[0014] The reinforcement learning scheme for recommended video acquisition based on the attraction and repulsion mechanism in this embodiment of the invention includes: Step 1, the topic module presets the acquisition topic and related keyword tags of the agent as the target guide for subsequent acquisition; Step 2, based on the target guide, the input module obtains video metadata from the video platform and converts it into an initial state vector; Step 3, the contrastive learning module performs unsupervised semantic modeling on the initial state vector, mapping multimodal content to a high-dimensional space to generate a topic vector for representing video semantics; Step 4, the state module dynamically integrates the features of the acquired video with the topic vector to form a feature vector describing the current environmental state; Step 5, the action module defines all interactive action spaces of the agent on the platform as an optional behavior set; Step 6, the Dueling DQN module estimates the expected Q value of each action in the action space in the current state based on the state feature vector; Step 7, the output module selects the optimal interactive action from the behavior set to perform acquisition based on the expected Q value, and calculates the reward value based on the multi-dimensional reward function of the attraction and repulsion mechanism, thereby updating the network parameters, wherein the reward value is used to guide the Q value estimation in subsequent steps.

[0015] The beneficial effects of this invention are as follows: Through the solution of this invention, a multi-dimensional reward function is designed. By simulating the attraction and repulsion mechanism in physics, the Agent focuses on collecting videos that match the theme category and characteristics of the internal video pool. It maintains the difference between the semantic gap and the video themes collected by other Agents, thereby achieving high coverage of recommended video data collection. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a reinforcement learning method for recommending video acquisition based on the attraction and repulsion mechanism, provided in an embodiment of the present invention. Figure 2 This is a technical architecture diagram of a recommendation video acquisition reinforcement learning method based on the attraction and repulsion mechanism provided in an embodiment of the present invention. Figure 3 This is a diagram of a video acquisition system based on a reinforcement learning method for recommending video acquisition using gravity and repulsion mechanisms, provided as an embodiment of the present invention. Detailed Implementation

[0018] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this invention, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0021] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0022] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0023] This invention provides a reinforcement learning method for recommending videos based on the attraction and repulsion mechanism, which can be applied to the video recommendation process in Internet scenarios.

[0024] See Figure 1This is a flowchart illustrating a reinforcement learning method for recommending video acquisition based on a gravity and repulsion mechanism, provided by an embodiment of the present invention. The method is applied to a reinforcement learning-based recommending video acquisition system, which includes a topic module, an input module, a contrastive learning module, a state module, an action module, a Dueling DQN module, and an output module. Figure 1 As shown, the method mainly includes the following steps: Step 1: The topic module presets the collection topics and related keyword tags of the intelligent agent as the target guide for subsequent collection. In specific implementation, such as Figure 2 As shown, its core innovation lies in the design of a multi-dimensional reward function. By simulating the gravitational and repulsive mechanisms in physics, the Agent focuses on collecting videos that match the theme categories and characteristics of its internal video pool. It maintains a difference in the theme of videos collected by other Agents by relying on semantic differences, thereby achieving high coverage of recommended video data collection.

[0025] The multi-dimensional reward function evaluates the effectiveness of actions based on category matching, feature matching, and semantic differences between the collected videos and the target video within the Agent.

[0026] (1) Category matching is used to limit the categories of content collected by the Agent, and is used to evaluate the degree of category matching between the content collected by the Agent and the existing content in its video pool. The categories of the existing content in the video pool reflect the Agent's interests. Similarity is calculated based on category matching. The category labels of all videos in the Agent's video pool are compiled into a category label list: ; in This indicates that the Agent collects the tag set of the first video in the Agent video pool, and represents the category tag of the target video as a category list C. For each element in the category list C... Calculate the number of times it appears in category list B, and divide by the total length of list B: ; in This represents the category label formed by the nth video under the current Agent. This represents the k-th category label of the current video content. This indicates that category k is in the category list. The number of times it appears in Represents a list of categories The length of the category list The number of categories in the category list C is denoted by n. The category matching score ranges from [0, 1]. The closer the category matching score is to 1, the higher the degree of category matching between the content collected by the Agent and the existing content in the video pool. Simultaneously, the category list... It should contain category labels for all known points of interest of the Agent, which can be applied when the target video category matches the current Agent with multiple points of interest.

[0027] (2) Semantic gap refers to the video collected by the current Agent. The theme differs from those of other Agent video pools.

[0028] The video collected by the current agent Category When the video v is fed into the encoder in the contrastive learning module, the feature vector of the video is obtained. : ; set up Represents an Agent category The video collection below, then the category The center position of the eigenvector It can be represented as: ; in Indicates the video category as The video in Indicates category The number of videos downloaded.

[0029] The feature vector of the input video Center position of the feature vector under each category Euclidean distance: ; Where m represents the dimension of the feature vector. and They represent and The value in the j-th dimension.

[0030] By treating video feature vectors as "particles," and simulating the attraction and repulsion mechanisms in physics, the attraction and repulsion effects between the current video and the agent's points of interest are quantified. The attraction is used to represent the driving force for the agent to collect videos that are highly related to existing points of interest, thereby strengthening the theme concentration. The repulsion force drives the agent to explore videos that are significantly different from existing interests, thereby improving long-tail coverage.

[0031] The current video feature vector is calculated using Gaussian function decay. Nearest Agent Feature Center The attraction between them G: ; in This is a scale parameter that controls the range of gravitational influence. The closer the distance, the greater the gravitational force, and vice versa, it approaches 0.

[0032] Calculate the current video feature vector using an inverse proportional function. With all other Agent topic feature centers The repulsive force P between them: ; in The repulsive force intensity coefficient, =1 This is a smoothing constant. The greater the distance, the smaller the contribution of the repulsive force, and vice versa.

[0033] Combining attraction and repulsion, the semantic gap is defined as: ; Gravitational weights control the intensity of interest focusing. =0.3 is the repulsive force weight, which adjusts the intensity of diversity exploration.

[0034] (3) Feature matching refers to the difference between the video features of the Agent’s internal video pool and the target video, including but not limited to interaction rate and video duration.

[0035] ; in , These are the normalized interaction rates of the target video and the normalized average interaction rates of the agent's video pool, respectively. , These are the target video duration binning process and the duration preference categories of the agent's video pool, respectively.

[0036] reward function To evaluate the merits of taking action a in state s, the reward function can be set according to the reward metric as follows: ; in , These are the weighted coefficients for category matching, semantic gap, and feature matching, respectively. In this way, the reward function can comprehensively consider the high coverage of the collected content (semantic gap) while the agent collects content according to its interests and requirements (category matching and feature matching), thus providing an effective reward mechanism for comprehensive data collection.

[0037] This invention is applied to a reinforcement learning-based video acquisition system for recommendation. The structure of the reinforcement learning-based video acquisition system for recommendation based on a multidimensional reward function is as follows: Figure 3 As shown: The topic module sets the topics to be collected, as well as a preset keyword library and tag list, in the early stages of Agent data collection. As the Agent continuously collects videos, it gradually focuses on topics that can earn higher rewards, thus forming stable points of interest. This module helps the Agent quickly establish personalized interest preferences, thereby improving the efficiency of subsequent comprehensive collection and recommendation of content.

[0038] Step 2: Based on goal orientation, the input module obtains video metadata from the video platform and converts it into an initial state vector; In practical implementation, this invention is applied to a reinforcement learning-based video acquisition system. The structure of the reinforcement learning-based video acquisition system based on a multidimensional reward function is as follows: Figure 3 As shown: 1) Input module: responsible for converting the collected video data into video feature state vectors, mainly including: video title, category set, interaction rate index and video duration, for subsequent model processing and calculation.

[0039] The input module first extracts the video title (T) and category tags from the video metadata, and then represents the tags as a list of categories C using strings: ; in This represents the k-th category label of the current video content.

[0040] The system obtains interactive data from video platforms, such as the number of likes, comments, and favorites, and calculates the interaction rate metric I. At the same time, it extracts the video playback duration from the video file and represents it as D.

[0041] .

[0042] Step 3: The contrastive learning module performs unsupervised semantic modeling on the initial state vector, mapping the multimodal content to a high-dimensional space to generate topic vectors for representing video semantics. In practice, the contrastive learning module maps video content to a high-dimensional feature space through unsupervised semantic modeling, generating topic vectors with semantic discriminative power. These vectors are used to quantify the semantic relationships between videos, providing core feature support for the attraction-repulsion mechanism in the reward function.

[0043] The module implements its functions through the following process: a. The encoder fuses the visual, textual, and interactive data of the video into a unified semantic vector, capturing multimodal features.

[0044] b. Through comparative learning, similar videos are clustered in the feature space (the distance between positive sample pairs is reduced), while dissimilar videos are separated (the distance between negative sample pairs is increased).

[0045] c. The generated topic vectors are input into the reward function to dynamically calculate the semantic gap (V), guiding the reinforcement learning agent to balance interest focus and diversity exploration.

[0046] Step 4: The status module dynamically integrates the features of the acquired video with the topic vector to form a feature vector describing the current environmental status. In practice, the status module integrates the video features collected by the Agent to form a feature vector that can comprehensively describe the Agent's current status. These features include video title, category label, interaction rate, and video duration. Through these features, the Agent's current status can be clearly understood, such as which categories of videos have been collected, the level of interaction of these videos, and the average duration of the videos. The Agent will choose an optimal action based on the current status, such as whether to collect a certain video.

[0047] state space Composed of the collected recommended content features T, C, I, and D, the system first performs expanded keyword matching on the video title to detect whether it contains topic-related keywords. The matching rate (TS) is calculated using a pre-set topic keyword library. For the video category tag, the tag matching degree (CS) is calculated by calculating the intersection ratio with the pre-set topic tags. The interaction rate index I is normalized to IS, and the video duration is converted to seconds and binned to DS. Videos shorter than 30 seconds are classified as short videos, longer than 30 seconds but shorter than 60 seconds as medium videos, and longer than 60 seconds as long videos. Therefore, the feature vector of state s can be represented as: .

[0048] Step 5: The action module defines the space of all interactive actions of the Agent on the platform, as an optional set of behaviors; In practical implementation, the action space is defined to include all possible actions that the model can perform in a given state. Since internet recommendation applications generally analyze content to be recommended based on user behavior, this invention defines such user behavior as actions, including actions such as liking, swiping, collecting, not collecting, saving, and commenting. Therefore, the action space A contains all possible actions that internet recommendations should be able to take, and action a can be represented as: ; in Let m be the i-th action, and m be the total number of actions.

[0049] Step 6: The Dueling DQN module estimates the expected Q-values ​​of each action in the action space to be performed in the current state based on the state feature vector. In practical implementation, the Dueling DQN module employs Dueling DQN as the reinforcement learning model. Dueling DQN is an improved deep Q-network that decomposes the Q-value into a state-value function and a dominance function. The state-value function evaluates the value of the current state, while the dominance function evaluates the advantage of each action relative to the average action. Dueling DQN better understands the value of states, especially when some actions have little impact on the outcome, thereby improving the accuracy and stability of the video acquisition strategy.

[0050] Assuming the current video state feature vector is s, DQN will use this feature vector to estimate the Q-value of each possible action taken in the current video state s. The specific calculation method is as follows: ; in This represents the Q value of taking action a in state s at time step t. Represents the state value function, Represents the dominance function. It indicates the size of the action space.

[0051] The network structure is shown in Table 1, mainly consisting of an input layer, hidden layers, branch layers, and an output layer. The input layer receives the video state feature vector. The hidden layer consists of two fully connected layers used to extract state features. The first layer has an input dimension of 128 and an output dimension of 64, with ReLU activation; the second layer has an input dimension of 64 and an output dimension of 32, with ReLU activation. The branching layer divides the output of the hidden layer into two branches: a state value function branch and a dominance function branch. The output layer combines the state value function and the dominance function to output the Q-value for each possible action.

[0052] Table 1 ; Step 7: The output module selects the optimal interactive action from the behavior set based on the expected Q value and performs the collection. It then calculates the reward value based on the multi-dimensional reward function of the attraction and repulsion mechanism and updates the network parameters. The reward value is used to guide the Q value estimation in subsequent steps.

[0053] In practical implementation, the output module selects an action based on the current state, collects video data, and uses... A strategy is used to select actions, which allows the model to explore to some extent. At each decision, a probability is used... Randomly select an action, to The probability of choosing the action with the maximum Q value is determined.

[0054] ; in The action is selected at time step t, based on the selected action. Perform corresponding actions, such as liking, swiping away, capturing, or not capturing video. The reward signal is calculated based on the action taken and the video capture results, reflecting the effect of the action. Let... The reward obtained at time step t. The next state reached after taking an action.

[0055] ; in, It is a discount factor. In the next state Take action below The Q-value is calculated by the target network. The parameters θ of the value network are updated by minimizing the loss function. The loss function is the mean squared error between the predicted Q-value and the target Q-value: ; Update the parameters of the value network using gradient descent: ; in, It is the learning rate, It is the loss function with respect to the parameters The gradient.

[0056] Furthermore, to improve the adaptability and robustness of the video acquisition system, a continuous learning mechanism is implemented by setting a playback buffer and periodically updating the target network. This allows the model to continuously update and improve as new data appears without forgetting previously learned knowledge. A buffer is set up and the target network is updated. The playback buffer is used to store experience tuples for random sampling training, which helps to break the correlation between data and stabilize the training process. The size of the buffer is set to... Store experience tuples The training process involves randomly selecting mini-batches of samples from a replay buffer. The parameters of the target network are periodically copied from the online network, which helps stabilize the training process. The update frequency formula is as follows: ; Where t is the current time step, and o is the update frequency. Setting o to 10000 means updating once every 10000 steps.

[0057] The recommendation video acquisition reinforcement learning method provided in this embodiment solves the information cocoon problem in recommendation systems caused by rule-based similar acquisition behaviors through a reinforcement learning-based simulated user behavior acquisition strategy. It constructs a video feature encoder from massive video data using contrastive learning technology, and obtains video topic vectors with discriminative power through the encoder, enabling the system to process massive amounts of data. It introduces a physical particle gravity and repulsion mechanism, which causes video features with the same topic to generate a gravity aggregation effect in the vector space, while heterogeneous video features form a repulsion isolation effect, realizing dynamic adaptive optimization of the feature space and breaking through the limitations of traditional static feature encoding.

[0058] The method of the present invention will be further described below with reference to a specific embodiment, which is to collect data from a video platform. The specific implementation steps are as follows: Step 1: The topic module presets the collection topics and related keyword tags for the Agent; The Agent is set to collect data on the theme of sports, with the following preset keywords: ['match', 'athlete', 'championship', 'sports meet', 'comeback']. The preset tag set is: {'track and field', 'sports meet', 'athlete', 'sports', 'basketball', 'football'}. Step 2: The input module obtains and processes video metadata from the video platform and converts it into an initial state vector; Metadata for a video collected from the Douyin platform is: {'span_content': 'My deskmate suddenly fainted on me (1) #catmeme #deskmate #campuslife #curiosity', 'video duration': '00:28', 'likes': '303,000', 'comments': '24,000', 'favorites': '37,000'} (1) Extract video title ; (2) Extract video tags ; (3) Calculate the interaction rate ; (4) Video duration .

[0059] Step 3: The contrastive learning module maps multimodal content to a high-dimensional space through unsupervised semantic modeling to generate topic vectors; (1) Encoder extracts features Input the video title, tags, and interaction data into the multimodal encoder to generate semantic vectors: ; (2) Calculate the attraction and repulsion forces ; ; ; Step 4: The status module dynamically integrates the collected features to form a feature vector describing the current environmental status. ; Step 5: The action module defines the space of all interactive actions of the Agent on the platform; ; ; ; ; Step 6: The Dueling DQN module estimates the expected Q-values ​​of each action taken in the current state based on the state feature vector; Input state vector Predict the Q-value using the Dueling DQN network: The ε-greedy strategy selects the action with the largest Q value with 90% probability and does not collect it, and randomly selects an action with 10% probability.

[0060] Step 7: The output module selects the optimal interaction action based on the Q value and updates the network parameters based on the multi-dimensional reward function of the attraction and repulsion mechanism; (1) Action execution If the strategy allows you to skip the current video and not collect any content, then skip this video.

[0061] (2) Experience storage and training Storage experience tuples ( ).

[0062] Conclusion: The current video "My deskmate suddenly fainted on me (1)" has a low overall reward value because the tag is not related to the sports theme, the semantic difference with other agents is small, the interaction rate and duration deviate from the agent's preference, and the system chooses not to collect it, so as to ensure that the agent focuses on the sports theme and gradually covers long-tail content.

[0063] This embodiment provides a reinforcement learning method for recommending videos based on a gravity and repulsion mechanism. By simulating real user behavior through reinforcement learning, it can adapt to the dynamic changes of recommendation algorithms in real time. Compared to traditional static keyword filtering, this invention, through the coupling of a "topic module" and a "contrastive learning module," enables the system to not only identify explicit keywords but also understand deep semantics. This ensures accurate positioning and focus on specific technical fields or topics within massive recommendation streams, avoiding "information bias" during the collection process. This invention innovatively introduces a "gravity and repulsion" reward mechanism based on physics principles. The gravity mechanism guides the agent to deeply mine content within the same topic, while the repulsion mechanism forces the agent to maintain differentiated exploration in the semantic space. This mechanism effectively breaks the "information cocoon" effect brought about by recommendation algorithms, enabling the collection of not only popular videos but also long-tail content and emerging topics that are difficult for traditional crawlers to reach, significantly improving data integrity. Due to the introduction of a multi-dimensional reward evaluation system, the system can predict action Q-values ​​through Dueling DQN before collection, filtering out videos that are irrelevant to the topic or highly repetitive from the source. This significantly reduces the pressure on backend storage space and the waste of bandwidth resources, improving the overall energy efficiency of the data collection task. Through regular updates of the buffer module and the target network, the system can continuously learn the latest distribution logic of the recommendation algorithm from the data collection experience. This means that as the data collection time increases, the system's judgment accuracy and collection efficiency will continuously improve, possessing a self-evolutionary characteristic that traditional tools do not have.

[0064] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof.

[0065] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A reinforcement learning method for recommending video acquisition based on attraction and repulsion mechanisms, the method being applied to a reinforcement learning-based recommending video acquisition system, the system comprising a topic module, an input module, a contrastive learning module, a state module, an action module, a Dueling DQN module, and an output module, characterized in that, The method includes: Step 1: The topic module presets the collection topics and related keyword tags of the intelligent agent as the target guide for subsequent collection. Step 2: Based on goal orientation, the input module obtains video metadata from the video platform and converts it into an initial state vector; Step 3: The contrastive learning module performs unsupervised semantic modeling on the initial state vector, mapping the multimodal content to a high-dimensional space to generate topic vectors for representing video semantics. Step 3 specifically includes: Step 3.1: The encoder fuses the visual, textual, and interactive data of the video into a unified semantic vector to capture multimodal features; Step 3.2: Through comparative learning, similar videos are clustered in the feature space, while dissimilar videos are separated. Step 3.3: Based on the generated topic vector, the semantic gap V is calculated using an attraction and repulsion mechanism. V is then used as a component of a multi-dimensional reward function to guide action evaluation. The semantic gap... The expression is: in, β is the gravitational weight, β is the repulsive weight, G is the attractive force, and P is the repulsive force. The expressions for calculating the attractive force G and the repulsive force P are as follows: in, The feature vector of the input video Nearest Agent Feature Center The Euclidean distance between them For scale parameters; The repulsive force intensity coefficient, It is a smoothing constant; Step 4: The status module dynamically integrates the features of the acquired videos to form a status feature vector, which includes video title matching rate, category matching degree, interaction rate index and video duration category. Step 5: The action module defines the space of all interactive actions of the Agent on the platform, as an optional set of behaviors; Step 6: The Dueling DQN module estimates the expected Q-values ​​of each action in the action space to be performed in the current state based on the state feature vector. Step 7: The output module selects the optimal interactive action from the behavior set based on the expected Q-value and performs the collection. It then calculates the reward value using a multi-dimensional reward function based on the attraction and repulsion mechanism. The semantic gap V in the reward function is calculated from the topic vector generated by the contrastive learning module, thereby updating the network parameters. The reward value guides Q-value estimation in subsequent steps. The multi-dimensional reward function evaluates the effect of taking action a in state s, and its expression is: in, , , These are the weighted coefficients for category matching, semantic discrepancy, and feature matching, respectively. For category matching degree, For semantic gap, This represents the feature matching degree.

2. The method according to claim 1, characterized in that, Category matching degree The expression is: in, This represents the k-th category label of the current video content. Indicate category The number of times it appears in category list B. Represents a list of categories Length, Represents a list of categories The number of categories.

3. The method according to claim 1, characterized in that, The Dueling DQN module decomposes the Q-value into a state-value function and a dominance function, and the expression for calculating the Q-value is as follows: in, This indicates that action a is taken in state s at time step t. value, Represents the state value function, Represents the dominance function. It indicates the size of the action space.

4. The method according to claim 3, characterized in that, The output module adopts Strategies for choosing actions.

5. The method according to claim 1, characterized in that, The method also includes a continuous learning step performed by the buffer module: A replay buffer is set up and the target network is updated periodically. The replay buffer is used to store empirical tuples. The parameters of the target network are periodically copied from the online network to stabilize the training process.

6. The method according to claim 1, characterized in that, The state feature vector consists of the following four components: the matching rate between the video title and the preset keyword tags, the matching degree between the video category tags and the collected topic, the normalized interaction rate index, and the category of the video duration after bucketing encoding.

Images