A live broadcast interaction enhancement method and device based on visual recognition

Abstract

The specification discloses a live broadcast interaction enhancement method and device based on visual recognition, comprising: in response to triggering of a card display event in a live broadcast process, acquiring a card image in a live broadcast video stream in real time; performing recognition processing on the card image to obtain card information including a type, a name and rarity information of the card; matching and querying the card information obtained through recognition with a preset card database to obtain corresponding basic value data, rarity data and heat index data; within a preset time window of card display, collecting barrage information in the live broadcast process, extracting keywords in the barrage, and analyzing emotional tendencies and appearance frequencies of the keywords; based on the basic value data, the rarity data, the heat index data and the barrage analysis result, calculating a comprehensive score of the card through a preset comprehensive scoring model; generating corresponding visual feedback content according to the comprehensive score, and superimposing the visual feedback content in the live broadcast picture in real time for display.

Description

Technical Field

[0001] This invention relates to the field of live streaming interactive technology, and more specifically, to a method and apparatus for enhancing live streaming interaction based on visual recognition. Background Technology

[0002] In recent years, with the booming development of the live streaming industry, card unpacking live streams have become a popular new form of live streaming. These live streams usually involve the host unpacking a physical card pack on the spot and showing the audience the contents of the cards obtained.

[0003] However, current interactive methods for such live streams mainly rely on traditional manual identification and explanation, which presents several significant problems: First, the streamer needs to manually identify the type, rarity, and value of the cards based on their personal knowledge and experience. This identification process is time-consuming and prone to errors, severely affecting the smoothness and rhythm of the live stream. Second, for viewers unfamiliar with specific card games, the lack of professional card knowledge makes it difficult to immediately understand the actual value and significance of the displayed cards, reducing their immersion and engagement. Furthermore, existing bullet screen interaction systems are limited in function and cannot provide personalized feedback based on the identified card information, failing to meet viewers' needs for in-depth interaction.

[0004] Therefore, providing a live streaming interaction enhancement method that can automatically identify card information, analyze audience feedback in real time, and generate personalized visual effects is of great practical significance for improving the professionalism and interactivity of card unpacking live streams.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] This specification provides a method and apparatus for enhancing live streaming interaction based on visual recognition, in order to overcome at least one technical problem existing in related technologies.

[0007] According to a first aspect of the embodiments of this specification, a method for enhancing live streaming interaction based on visual recognition is provided, comprising: In response to the card display event during the live stream, card images from the live video stream are acquired in real time; The card image is processed for recognition to obtain card information including card type, name, and rarity information; The identified card information is matched and queried with a preset card database to obtain the corresponding basic value data, rarity data and popularity index data; Within a preset time window for card display, collect bullet screen information during the live stream, extract keywords from the bullet screen, and analyze the emotional tendency and frequency of occurrence of the keywords. Based on the aforementioned basic value data, rarity data, popularity index data, and bullet screen analysis results, the card's overall score is calculated using a preset comprehensive scoring model. Based on the comprehensive score, corresponding visual feedback content is generated and superimposed onto the live broadcast screen in real time for display.

[0008] In some optional implementations, the real-time acquisition of card images from the live video stream includes: Obtain real-time video data through the video stream interface provided by the live streaming platform; The real-time video data is analyzed using an object detection algorithm. The frame analysis includes: processing the video frames using an object detection model, detecting card targets in the video frames using the object detection model, and outputting the coordinate information of the card area, extracting the region of interest of the card based on the coordinate information, and preprocessing the region of interest, including size normalization and color space conversion. The detected card area is subjected to image enhancement processing, which includes multi-frame fusion and edge optimization; The processed card image is output as the image to be recognized.

[0009] In some optional implementations, the recognition processing of the card image includes: The card images are classified using a deep neural network model to identify the series category to which the cards belong; Based on the classification results, an appropriate recognition strategy is selected. For cards with clear text information, optical character recognition technology is used to extract key information; for cards with obvious features, image feature matching algorithm is used for comparison. Based on the combined classification results and identification information, the type, name, and rarity information of the cards are determined; The deep neural network model adopts a convolutional neural network architecture and is fine-tuned using card image data based on the pre-trained model; the optical character recognition uses an OCR engine and combines post-processing logic to remove background interference; the image feature matching algorithm selects either the scale-invariant feature transformation algorithm or the fast rotation-invariant feature algorithm to solve the recognition problem under conditions of card rotation, scaling, and partial occlusion; and the best matching result is selected through a confidence scoring mechanism.

[0010] In some optional implementations, the calculation formula for the comprehensive scoring model is as follows: Overall score = α × Normalized base value + β × Rarity value + γ × Popularity index + θ × Comment score Among them, symbols , , and The normalized basic value is calculated by log normalization of the basic value, which is a preset weight coefficient. The popularity index is calculated by normalization based on monthly search volume or platform mentions. The bullet screen score is calculated by the weight and frequency of bullet screen keywords. The calculation process for the bullet screen rating includes: Establish a dictionary containing emotional keywords and evaluative keywords, with each keyword having a preset emotional value and weight factor; count the frequency of each keyword within a set time window; sum the products of the weight factor and the frequency of occurrence, and then divide by the total number of bullet comments within the time window to obtain the bullet comment rating.

[0011] In some optional implementations, generating the corresponding visual feedback content includes: The feedback level is determined based on the comprehensive score, and different feedback levels correspond to different visual effects. Visual elements are generated using a graphics rendering engine, including star ratings, light effects animations, and text information. The visual elements are overlaid onto the live stream output screen through the plugin interface of the live streaming software.

[0012] In some optional implementations, the method further includes exception handling and user intervention steps, specifically including: When the confidence level of card recognition falls below a set threshold, a recognition assistance request is sent to the audience. Collect audience feedback as training data; Optimize the recognition model using user feedback data based on supervised learning; Record user evaluation data on the recognition results, including error correction information and rating voting results.

[0013] In some optional implementations, the method further includes a system performance optimization step, specifically including: The identification service is deployed using a distributed computing architecture, and processing efficiency is improved through load balancing. Prioritize and schedule recognition tasks to keep the delay in real-time feedback within a set threshold. Low-confidence samples were manually labeled using an active learning strategy. By introducing a misjudgment scenario construction technique, the model's ability to distinguish similar cards is improved.

[0014] In some optional implementations, the method further includes a multi-card game adaptation step, specifically including: Provides a card template upload interface, supporting the addition of new card types; Establish independent recognition sub-models and databases for different card games; Automatically select the corresponding recognition model and database based on live stream tag information; By using transfer learning, the pre-trained model is fine-tuned using new card data to adapt to new card types.

[0015] In some optional implementations, the method further includes steps for building and updating a card database, specifically including: Basic card information was collected from multiple third-party card platforms using web crawling technology; Regularly update the market valuation and popularity data of the cards; Establish independent database subsets for different card games; Based on the tag information of the live broadcast content, the system automatically switches to the corresponding database subset for query matching.

[0016] According to a second aspect of the embodiments of this specification, a live interactive enhancement device based on visual recognition is provided, comprising: The image acquisition module is used to acquire card images from the live video stream in real time in response to the card display event during the live broadcast. The image recognition module is used to recognize and process the card image to obtain the card type, name, and rarity information; The data matching module is used to match and query the identified card information with the preset card database to obtain the corresponding basic value data, rarity data and popularity index data. The bullet screen analysis module is used to collect bullet screen information during the live broadcast within a preset time window of card display, extract keywords in the bullet screen, and analyze the sentiment and frequency of the keywords. The comprehensive scoring module is used to calculate the card's comprehensive score based on the basic value data, rarity data, popularity index data, and bullet screen analysis results, using a preset comprehensive scoring model. The visualization feedback module is used to generate corresponding visual feedback content based on the comprehensive score, and to overlay the visual feedback content onto the live broadcast screen in real time for display.

[0017] The beneficial effects of the embodiments in this specification are as follows: The visual recognition-based live streaming interaction enhancement method disclosed in this application involves: responding to the triggering of card display events during the live stream by acquiring card images from the live video stream in real time; processing the card images to obtain card information; matching the recognition results with a preset card database to obtain corresponding basic data; collecting and analyzing the sentiment and frequency of keywords in the bullet screen information; calculating the comprehensive card score based on multi-dimensional data using a comprehensive scoring model; and generating visual feedback content based on the scoring results and overlaying it onto the live stream screen in real time. On the one hand, replacing manual recognition with automated image recognition technology reduces the cognitive burden and operational threshold for the streamer, improves the accuracy and efficiency of card information recognition, and ensures the smoothness of the live stream process. On the other hand, combining the card's basic attributes with real-time bullet screen sentiment analysis for multi-dimensional comprehensive evaluation can more comprehensively and objectively reflect the card's actual value and audience expectations, enhancing the scientific nature and credibility of the value assessment. Furthermore, by overlaying the analysis results onto the live stream screen in a visual form in real time, it can provide viewers with intuitive value references and an immersive visual experience, effectively solving the comprehension barriers for non-professional viewers and enhancing the depth and engagement of the interaction.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments or related technologies of this specification, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 The illustration shows a schematic diagram of the overall flow of a visual recognition-based live interaction enhancement method according to some embodiments of the present disclosure; Figure 2 The illustration shows a schematic diagram of a method flow for acquiring and preprocessing card images in a live video stream according to some embodiments of the present disclosure; Figure 3 The illustration shows a schematic diagram of a multi-strategy card image recognition processing method according to some embodiments of the present disclosure; Figure 4 The illustration shows a schematic diagram of a card comprehensive score calculation method flow according to some embodiments of the present disclosure; Figure 5 The illustration shows a schematic diagram of a method for generating card visual feedback content and overlaying live video according to some embodiments of the present disclosure; Figure 6 This schematic diagram illustrates the functional module structure of a visual recognition-based live interactive enhancement device according to some embodiments of the present disclosure; Figure 7 The illustration shows a schematic diagram of information interaction among the modules in a visual recognition-based live interactive enhancement device according to some embodiments of the present disclosure. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of one or more embodiments of this specification clearer, the technical solutions of one or more embodiments of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of one or more embodiments of this specification.

[0022] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another.

[0023] In this example embodiment, a method for enhancing live streaming interaction based on visual recognition is provided. This method is applicable to live streaming scenarios such as card unpacking. The method can be deployed in a distributed system composed of cloud servers, edge computing nodes, and broadcaster terminal devices. By integrating computer vision, natural language processing, and real-time rendering technologies, it provides intelligent interactive enhancement functions for live streaming. Figure 1 A schematic diagram illustrating a live streaming interaction enhancement method based on visual recognition according to an embodiment of the present disclosure is shown. (Refer to...) Figure 1 The method includes the following steps: In step S110, in response to the triggering of the card display event during the live broadcast, the card images in the live video stream are acquired in real time; In step S120, the card image is processed for recognition to obtain the card type, name, and rarity information; In step S130, the identified card information is matched and queried with a preset card database to obtain the corresponding basic value data, rarity data and popularity index data; In step S140, within a preset time window for card display, bullet screen information during the live stream is collected, keywords in the bullet screen are extracted, and the emotional tendency and frequency of occurrence of the keywords are analyzed. In step S150, based on the basic value data, rarity data, popularity index data, and bullet screen analysis results, the card's comprehensive score is calculated using a preset comprehensive scoring model. In step S160, corresponding visual feedback content is generated based on the comprehensive score, and the visual feedback content is superimposed onto the live broadcast screen in real time for display.

[0024] The technical solution of this embodiment can replace the traditional method of relying on the anchor to manually identify cards through automated image recognition technology, thereby improving the pace of live streaming and recognition efficiency. Simultaneously, it combines the objective attribute data of the cards with the subjective emotional reactions of the audience's comments, using a scientific scoring model to conduct a comprehensive value assessment and generate corresponding dynamic visual feedback. This technical solution not only effectively lowers the cognitive threshold for viewers but also enhances their sense of participation and immersion through personalized interactive responses, thereby improving the overall quality and professionalism of live streaming interaction.

[0025] The steps of this embodiment will be described in detail below.

[0026] In step S110, in response to the triggering of the card display event during the live broadcast, the card images in the live video stream are acquired in real time.

[0027] In this embodiment, the card display event refers to the visual moment when the broadcaster begins to display a specific card in the live stream. The system accesses real-time video stream data through the application programming interface (API) provided by the live streaming platform (such as OBSStudio, XSplit, etc.). To accurately capture card images, a deep learning-based object detection algorithm can be used to analyze the video stream frame by frame. Specifically, a YOLOv5 object detection model trained on a large-scale card image dataset is used. This model can identify and locate card regions in video frames with high accuracy and speed, and output the bounding box coordinates of the card regions.

[0028] Based on the coordinate information, the system extracts the Region of Interest (ROI) of the card from the original video frames. To improve the accuracy of subsequent recognition processing, a series of preprocessing operations can be performed on the extracted ROI, including standardizing the image size to a uniform specification (e.g., 224×224 pixels) and performing color space conversion (e.g., from BGR to RGB format). Considering factors common in live streaming environments such as hand tremors and lighting changes, image enhancement processing can also be applied to the detected card area. Specifically, this can include multi-frame fusion technology to eliminate motion blur and edge optimization algorithms (e.g., Canny edge detection combined with morphological operations) to enhance the clarity of the card outline. Finally, the enhanced and clear card image is output as the image to be recognized to the recognition processing module.

[0029] In step S120, the card image is processed to obtain the card type, name, and rarity information.

[0030] The recognition and processing stage can employ a multi-strategy adaptive recognition scheme. The details of this scheme are explained below. First, a deep convolutional neural network model based on the ResNet-50 architecture can be used to classify card images. This model was pre-trained on the large ImageNet visual dataset and further fine-tuned using over 100,000 images covering various mainstream card games to accurately identify the series or category to which the cards belong, such as Pokémon, Yu-Gi-Oh!, and Magic: The Gathering. Based on the initial classification results, the optimal subsequent recognition path can be intelligently selected. Specifically, for cards with clearly printed text and a standardized layout, optical character recognition (OCR) technology based on the PaddleOCR engine can be used to extract the name, number, and rarity markers (such as "SR", "UR", "SSR", etc.) from the cards. The OCR processing incorporates specially designed post-processing logic, including removing complex background interference through morphological filtering and connected component analysis, as well as contextual semantic error correction based on card naming rules, ensuring the accuracy of text recognition. For cards with unique visual features, or when OCR recognition is ineffective due to special fonts, poor lighting, or other reasons, image feature matching algorithms can be used. Specifically, SIFT (Scale Invariant Feature Transform) or ORB (Rotation Invariant Feature Transform) algorithms are selected. These algorithms have excellent robustness to image rotation, scaling, and partial occlusion. They can extract and match feature points between the card image to be identified and a large number of pre-stored card templates in a database, and determine the unique identity of the card by calculating the similarity of feature descriptors.

[0031] Finally, a confidence scoring mechanism can be introduced to quantitatively evaluate and integrate results from different recognition paths. For example, when the OCR confidence level for rarity marker recognition reaches 96%, while the confidence level for feature matching for overall card identity confirmation is 90%, the proposed solution will prioritize the OCR result with higher confidence and comprehensively output the card's final type, name, and rarity information. Testing shows that the Top-1 accuracy of this integrated recognition scheme is consistently above 95.3%, and the Top-3 accuracy reaches 98.1%.

[0032] In step S130, the identified card information is matched and queried with a preset card database to obtain the corresponding basic value data, rarity data and popularity index data.

[0033] This application's technical solution maintains a dynamically updated, structured core card database. This database continuously collects basic card information from multiple authoritative third-party card trading and information platforms, such as TCGPlayer and official card game websites, using efficient distributed web crawler technology. The database schema design can include key fields such as card number, name, series, rarity value (RarityValue), market benchmark valuation (BaseValue), issuance volume, search volume in the past 30 days, and mentions on mainstream social media platforms. All data is automatically updated daily to ensure the timeliness of market valuations and popularity indices.

[0034] Once the card is successfully identified, the technical solution of this application uses its name and series as key indexes to perform a precise query in the database to obtain its standardized basic value data, internally defined rarity data (usually normalized values), and a popularity index (HotIndex) that combines monthly search volume and platform mentions and is obtained through max-min normalization, with a value range of 0 to 1.

[0035] In step S140, within a preset time window for card display, bullet screen information during the live stream is collected, keywords in the bullet screen are extracted, and the emotional tendency and frequency of occurrence of the keywords are analyzed.

[0036] In order to capture the audience's immediate collective reaction to the currently displayed card, the technical solution of this application collects and receives all the bullet screen data streams in the live broadcast room in real time within a preset time window after the card is successfully recognized, typically set to 30 seconds.

[0037] The technical solution of this application can incorporate a meticulously constructed professional keyword dictionary, which categorizes keywords into two types: emotional keywords and evaluative keywords. Emotional keywords, such as "god-tier card," "trash," "must-upgrade," and "huge loss," are primarily used to capture and quantify the audience's emotional inclinations. Evaluative keywords, such as "gold card," "SSR," "limited edition," and "flash card," are mainly used to describe the objective attributes or rarity characteristics of the cards. Each keyword is pre-assigned an emotional value (e.g., +1 for positive emotion, -1 for negative emotion, and 0 for neutral) and a weighting factor (set according to the importance of the word in the card evaluation context; for example, "SSR" may have a higher weight than "normal"). This technical solution can statistically analyze the frequency of each keyword within a given time window and record the total number of bullet comments during that time period, providing a data foundation for subsequent calculation of bullet comment scores.

[0038] In step S150, based on the basic value data, rarity data, popularity index data, and bullet screen analysis results, the card's comprehensive score is calculated using a preset comprehensive scoring model.

[0039] This embodiment quantitatively integrates the static objective attributes of the cards with the dynamic subjective emotions of the community. The model can be implemented using the following weighted linear formula: Overall score = α × Normalized base value + β × Rarity value + γ × Popularity index + θ × Comment score The normalized base value (BaseValue_norm) is normalized by taking the base-10 logarithm of the card's market benchmark valuation (BaseValue) and then dividing it by the logarithm of the highest benchmark valuation in the dataset. That is, BaseValue_norm = log0 10 (BaseValue) / log 10 (BaseValue_max) This processing aims to smooth out potentially large price differences between different cards, making them more consistent with a linear model. The HotIndex, calculated using a max-min normalization method based on the card's monthly search volume and platform mentions, has a value range of [0,1] and reflects the card's recent market attention. The calculation process for the bullet screen score is as follows: First, for each keyword appearing within the time window, calculate the product of its weight factor (w_i) and frequency of occurrence (f_i). Then, sum the results of this product for all keywords and finally divide by the total number of bullet screens N within the time window, i.e., Bullet Screen Score = (Σw_i×f_i) / N. This method can effectively quantify the collective emotional intensity of the audience while avoiding bias caused by fluctuations in the total number of bullet screens.

[0040] The weighting coefficients α, β, γ, and θ are preset weighting coefficients used to balance the influence of various factors on the final score. In a preferred configuration, the default values ​​of α=0.3, β=0.3, γ=0.2, and θ=0.2 can be used. It should be noted that these coefficients can be flexibly adjusted and optimized according to the user characteristics of different live streaming platforms or the ecosystem of a specific card game.

[0041] In step S160, corresponding visual feedback content is generated based on the comprehensive score, and the visual feedback content is superimposed onto the live broadcast screen in real time for display.

[0042] In the technical solution of this application, the calculated comprehensive score (usually quantified as 0-10 points) can be mapped to a preset feedback level. The specific level division can be as follows: 9-10 points: Legendary - Triggers a full-screen golden particle light effect, dynamically rotating star indicators (such as five gold stars), accompanied by exciting exclusive sound effects, and displays detailed text information.

[0043] 7-8 points: Rare - Triggers a blue spectrum halo around the card and a star-level animation that lights up one star at a time (e.g., three to four stars).

[0044] 5-6 points: Normal level - Displays color-coded border effects and basic text labels (name, estimated value).

[0045] 1-4 points: Common level - The core information of the card is displayed only in a simple floating text box.

[0046] The aforementioned visual elements, including dynamic star ratings, particle effects, animations, and structured text information (such as card names, real-time market valuations, popularity levels, and rarity indicators), can be generated in real-time by high-performance graphics rendering engines such as Unity3D or WebGL. Subsequently, through a dedicated plugin interface developed for mainstream live streaming software such as OBS Studio or XSplit, these rendered visual feedback contents with alpha channels are precisely and with low latency composited into the output of the live stream using a layer overlay method, ultimately presented to all online viewers.

[0047] The visual recognition-based live streaming interaction enhancement method disclosed in this application involves: responding to the triggering of card display events during the live stream by acquiring card images from the live video stream in real time; processing the card images to obtain card information; matching the recognition results with a preset card database to obtain corresponding basic data; collecting and analyzing the sentiment and frequency of keywords in the bullet screen information; calculating the comprehensive card score based on multi-dimensional data using a comprehensive scoring model; and generating visual feedback content based on the scoring results and overlaying it onto the live stream screen in real time. On the one hand, replacing manual recognition with automated image recognition technology reduces the cognitive burden and operational threshold for the streamer, improves the accuracy and efficiency of card information recognition, and ensures the smoothness of the live stream process. On the other hand, combining the card's basic attributes with real-time bullet screen sentiment analysis for multi-dimensional comprehensive evaluation can more comprehensively and objectively reflect the card's actual value and audience expectations, enhancing the scientific nature and credibility of the value assessment. Furthermore, by overlaying the analysis results onto the live stream screen in a visual form in real time, it can provide viewers with intuitive value references and an immersive visual experience, effectively solving the comprehension barriers for non-professional viewers and enhancing the depth and engagement of the interaction.

[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0049] Figure 2 The illustration schematically depicts a flow diagram of an image acquisition and preprocessing method according to some embodiments of the present disclosure. Reference Figure 2 As shown, the method may include the following steps: In step S210, real-time video data is obtained through the video stream interface provided by the live streaming platform; In step S220, a target detection algorithm is used to process the video frame, detect the card target, and output its coordinate information; In step S230, the region of interest of the card is extracted based on the coordinate information; In step S240, the region of interest is preprocessed, including size normalization and color space conversion; In step S250, multi-frame fusion and edge optimization are performed on the preprocessed image; In step S260, the processed card image is output as the image to be recognized.

[0050] The above steps allow us to obtain high-quality card images, providing a solid foundation for subsequent recognition and processing. Each step has its specific function, collectively ensuring the quality and efficiency of image acquisition and preprocessing.

[0051] Figure 3 A schematic diagram illustrating a card recognition processing method flow according to some embodiments of the present disclosure is provided. (Reference) Figure 3 As shown, the method may include the following steps: In step S310, a deep neural network model is used to classify the card images and identify the series to which the cards belong; In step S320, it is determined whether the text information on the card is clear; In step S330, if so, key information is extracted using OCR technology; In step S340, if not, an image feature matching algorithm is used for comparison; In step S350, the detailed information of the card is determined by combining the classification results and the identification information.

[0052] The following is a detailed description of this embodiment. In step S310, a deep convolutional neural network model based on the ResNet-50 architecture can be used to classify the preprocessed card images. This model is pre-trained on the large ImageNet visual dataset and can be further fine-tuned using multiple image datasets covering various mainstream card games such as Pokémon, Yu-Gi-Oh!, Magic: The Gathering, and Shadowverse. Through multi-level convolution and pooling operations, the model can automatically learn and extract high-level features from the card images, such as art style, layout design, and border features, thereby accurately identifying the series category to which the cards belong.

[0053] In step S320, the technical solution of this application can perform a sharpness analysis on the text regions in the card image using an image quality assessment algorithm, mainly detecting key indicators such as contrast, sharpness, and noise level of the text regions. During the assessment process, the algorithm automatically locates the text regions on the card (such as card name, number, rarity marker, etc.) and calculates the sharpness score of these regions. When the score is higher than a preset threshold, the text information is determined to be clear, and the system proceeds to step S330; otherwise, the text information is determined to be blurry or severely occluded, and the system proceeds to step S340.

[0054] In step S330, for cards with clear text information, key information can be extracted using optical character recognition technology based on the PaddleOCR engine. This OCR engine combines deep learning algorithms and post-processing logic, effectively removing interference from complex background patterns and accurately recognizing text content in various fonts and layout formats. The specific processing flow may include: first, binarizing the text area; then, using a text detection algorithm to locate the character position; next, converting it into text information through a character recognition model; and finally, using post-processing logic for error correction and format standardization, ultimately extracting key information such as the card's name, number, and rarity markers (e.g., "SR", "UR", "SSR").

[0055] In step S340, for cards with unclear text information, image feature matching algorithms can be used for comparison and recognition. Specifically, SIFT (Scale Invariant Feature Transform) or ORB (Rotation Invariant Feature Transform) algorithms can be selected, as these algorithms have excellent robustness to image rotation, scaling, lighting changes, and partial occlusion. During processing, a large number of local feature points are first extracted from the image of the card to be identified, and unique feature descriptors are generated. Then, these feature descriptors are compared with the feature descriptors of the corresponding series of card templates pre-stored in the database, a similarity score is calculated, and finally, the template with the highest similarity is selected as the matching result.

[0056] In step S350, the series classification results from step S310 and the identification information from steps S330 or S340 are combined, and a multi-source information fusion algorithm is used to determine the detailed information of the card. During the fusion process, information from different sources is assigned corresponding confidence weights, and a comprehensive decision is made using methods such as Bayesian inference or DS evidence theory. Specifically, this scheme uses the series classification results as a filtering condition, performs precise matching in the corresponding series' database subset, and combines the text information extracted by OCR or the results of feature matching, using a confidence scoring mechanism to select the best match. The final output detailed information includes the card's complete name, accurate serial number, series, rarity level, release year, and other comprehensive card attribute information.

[0057] By following the steps above, the most suitable recognition strategy can be selected based on the different characteristics of the cards, improving the accuracy and efficiency of recognition. This adaptive recognition method can adapt to different types of cards, improving the system's versatility.

[0058] Figure 4 A schematic diagram illustrating the flow of a comprehensive score calculation method according to some embodiments of the present disclosure is provided. (Reference) Figure 4 As shown, the method may include the following steps: In step S410, the card's base value, rarity value, and popularity index are obtained from the database; In step S420, the basic value is logarithmically normalized; In step S430, bullet screen information within a preset time window is collected; In step S440, keywords in the bullet comments are extracted and their frequency of occurrence is counted; In step S450, the bullet screen score is calculated; In step S460, the comprehensive score is calculated according to the formula.

[0059] By taking the above steps into account, multiple factors can be considered to obtain a comprehensive card value score. This multi-factor scoring method can more accurately reflect the actual value and popularity of a card.

[0060] Figure 5 A schematic diagram illustrating a visual feedback generation method flow according to some embodiments of the present disclosure is provided. (Reference) Figure 5 As shown, the method may include the following steps: In step S510, the feedback level is determined based on the comprehensive score; In step S520, select the corresponding visual effects template; In step S530, star ratings and light effect animations are generated; In step S540, card information text is generated; In step S550, visual elements are overlaid onto the live stream screen; In step S560, the processed live broadcast image is output.

[0061] The following is a detailed description of this embodiment. In the visual feedback generation and overlay stage, the technical solution of this application can divide the comprehensive score of 0-10 into four levels, namely, 9-10 points triggers "legendary" feedback, 7-8 points is "rare", 5-6 points is "normal", and 1-4 points is "common". Each level not only corresponds to different visual effect intensities, but also allows for the setting of a dynamic transition mechanism. That is, when the score is at the level threshold, a hybrid rendering technique is used to achieve a smooth transition, avoiding abrupt changes in visual performance.

[0062] In the special effects generation stage, this application's technical solution employs physically based rendering technology to construct visual elements. For the generation of star-level symbols, a parametric particle system can be developed. By controlling the initial velocity, lifespan, and color gradient of the particle emitters, a golden starburst effect spreading outward from the center can be achieved. The lighting animation utilizes real-time ray tracing technology, generating a surrounding halo by calculating the normal vectors of the card's edges, and dynamically adjusting the halo radius and color saturation based on the rating. Specifically, for legendary cards, a full-screen particle diffusion effect based on fluid dynamics simulation can be introduced, achieving the natural movement of the smoke-like effect by solving the Navier-Stokes equations. Text information generation can employ a multi-layer rendering architecture. The bottom layer uses Signed Distance Fields technology to ensure the text remains clear and sharp at any resolution, while the upper layer automatically extracts key information and optimizes the layout through semantic analysis. The technical solution of this application uses an adaptive layout engine to intelligently adjust the text display position according to the feature area of ​​the card image, avoiding obscuring key visual elements of the card. All visual elements are precisely controlled through deep buffer management to ensure the perfect integration of special effects, text and the original live broadcast screen.

[0063] In the image compositing and output stage, the technical solution of this application can adopt hardware-accelerated alpha blending technology to achieve seamless overlay of visual elements and live screen by calculating the transparency channel of each pixel. For different streaming platforms, multi-channel output adapters are developed to support real-time conversion of multiple color spaces such as RGB and YUV. Simultaneously, a timestamp synchronization mechanism ensures precise alignment between visual feedback and the video stream. Finally, the video stream is compressed by an H.265 encoder and output to the live streaming platform. End-to-end processing latency is strictly controlled to an extremely short time, such as within 33 milliseconds, thereby ensuring viewers receive an immediate visual feedback experience.

[0064] By following the steps above, viewers can be provided with intuitive and vivid visual feedback, enhancing the interactivity and watchability of the live stream. This real-time visual feedback improves the viewing experience and increases the appeal of the live stream.

[0065] In one example embodiment of this disclosure, the method may further include anomaly handling and user participation steps. Specifically, when the confidence score output by the image recognition module is lower than a set reliability threshold (e.g., 70%), it indicates that the automatic recognition of the current card by the technical solution of this application is uncertain. At this time, an interactive recognition assistance request can be automatically initiated to the audience in the live broadcast interface, such as through a prominent pop-up window for multi-option voting. The collected audience feedback information (such as majority voting results, text-based error correction comments) is labeled as high-quality labeled data and used for incremental training and optimization of the subsequent recognition model based on a supervised learning paradigm, thereby achieving the self-evolution of the model. At the same time, the technical solution of this application can establish a complete user behavior log, recording all relevant evaluation data, including error correction records and rating votes, thereby forming a continuous learning closed loop of "use-feedback-optimization".

[0066] In one example embodiment of this disclosure, the method may further include performance optimization steps. Specifically, to meet the stringent real-time requirements of live streaming scenarios, the core recognition and scoring service can adopt a Kubernetes-based containerized microservice architecture, deployed on a cloud server cluster with GPU acceleration capabilities, and dynamically allocate computing tasks through an intelligent load balancer. The technical solution of this application can implement dynamic priority scheduling for concurrent recognition tasks and utilize high-performance memory data storage such as Redis as an intermediate message queue and cache intermediary to ensure that the end-to-end processing latency from the appearance of the card on the screen to the completion of the final visual feedback rendering is strictly controlled within 300 milliseconds.

[0067] For continuous model optimization, an active learning strategy can be adopted to automatically filter out samples with low identification confidence and prioritize them for the manual annotation pipeline, thereby efficiently expanding the high-quality training dataset. In addition, Hard Negative Mining technology can be introduced to construct adversarial training scenarios that are prone to misjudgment and have similar features, continuously improving the model's discriminative ability in fine-grained classification.

[0068] In one example embodiment of this disclosure, the method further includes a multi-card game adaptation step.

[0069] The system designed according to the technical solution of this application supports a highly pluggable extension mechanism. This allows third-party developers of emerging or niche card games to easily upload custom card image templates and related metadata through the system's standardized RESTful API interface. At the underlying level, the system can establish completely independent recognition sub-models and database subsets for different games such as Pokémon, Yu-Gi-Oh!, Magic: The Gathering, and Shadowverse, thus achieving data and model isolation. During the live streaming initialization phase, the system can automatically route and load the corresponding recognition model and database subset for processing based on the streamer's preset live room tags or through rapid preliminary analysis of the first frame image.

[0070] For newly released card series, transfer learning techniques can be used to quickly fine-tune existing, well-generalized pre-trained models (such as base models trained on ImageNet and mixed card data) using a relatively small amount (e.g., thousands) of new card image data. This usually achieves the accuracy required for production environments within a few hours, realizing efficient and low-cost adaptation to new card types.

[0071] In one example embodiment of this disclosure, the method further includes the steps of constructing and updating a card database. Specifically, the system designed according to the technical solution of this application can collect basic card information from multiple third-party card platforms such as TCGPlayer and card game websites using distributed web crawler technology. A daily automatic update mechanism is established to regularly update the market valuation and popularity data of cards, ensuring the timeliness of the data. Independent database subsets are established for different card games such as Pokémon, Yu-Gi-Oh!, and Magic: The Gathering. Based on the tag information and preliminary identification results of the live broadcast content, the corresponding database subset is automatically switched for accurate query matching.

[0072] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0073] Furthermore, in this example embodiment, a live streaming interaction enhancement device based on visual recognition is also provided. (See reference...) Figure 6 As shown, the device 600 includes: an image acquisition module 610, an image recognition module 620, a data matching module 630, a bullet screen analysis module 640, a comprehensive scoring module 650, and a visualization feedback module 660. Wherein: The image acquisition module 610 is used to acquire card images in the live video stream in real time in response to the triggering of card display events during the live broadcast. The image recognition module 620 is used to recognize and process the card image to obtain the card type, name, and rarity information; The data matching module 630 is used to match and query the identified card information with the preset card database to obtain the corresponding basic value data, rarity data and popularity index data. The bullet screen analysis module 640 is used to collect bullet screen information during the live broadcast within a preset time window of card display, extract keywords in the bullet screen, and analyze the sentiment tendency and frequency of occurrence of the keywords. The comprehensive scoring module 650 is used to calculate the comprehensive score of the card based on the basic value data, rarity data, popularity index data and bullet screen analysis results, through a preset comprehensive scoring model. The visualization feedback module 660 is used to generate corresponding visual feedback content based on the comprehensive score, and to overlay the visual feedback content onto the live broadcast screen in real time for display.

[0074] like Figure 7 As shown, Figure 7 This is a schematic diagram illustrating the information interaction between the various modules in the device, such as... Figure 7As shown, the interaction process of each module can be as follows: After the system starts, the control and management module, acting as the coordination hub, first receives real-time image data from the live video stream and distributes it to the image acquisition module for processing. The image acquisition module extracts the card ROI region using a target detection algorithm, and then transmits the preprocessed card image to the image recognition module. The image recognition module uses a deep learning model to classify and extract features from the cards, identifies the basic information of the cards, and feeds the results back to the control and management module. Simultaneously, the control and management module coordinates the data matching and scoring module to obtain information such as the basic value, rarity, and popularity index of the corresponding cards from the card database. In the parallel processing path, the bullet screen analysis module receives external bullet screen input, extracts keywords and analyzes sentiment trends using natural language processing technology, and also transmits the bullet screen scoring results to the data matching and scoring module. The data matching and scoring module integrates the image recognition results and bullet screen analysis data, uses a preset algorithm to calculate the comprehensive score of the card, and sends the scoring results to the visualization feedback module. The visualization feedback module generates corresponding star ratings, lighting animations, and text information based on the comprehensive score. These visual elements are then precisely overlaid onto the original live stream using layer compositing technology. Finally, the enhanced live stream is output to the viewer interface via the live streaming platform interface, providing real-time visual feedback. Throughout the entire interaction process, the control and management module continuously monitors the operational status of each module, ensuring real-time data transmission and smooth processing flow. The end-to-end latency from card appearance to feedback display is controlled within 300 milliseconds.

[0075] In some example embodiments of this disclosure, based on the foregoing scheme, the image acquisition module 610 includes: The video stream acquisition unit is used to acquire real-time video data through the video stream interface provided by the live streaming platform. The target detection unit is used to perform frame analysis on the real-time video data using a target detection algorithm to locate the card area in the video frame. An image enhancement unit is used to perform image enhancement processing on the detected card area, the image enhancement processing including multi-frame fusion and edge optimization; The image output unit is used to output the processed card image as the image to be recognized.

[0076] In some example embodiments of this disclosure, based on the foregoing scheme, the image recognition module 620 includes: A series classification unit is used to classify the card images using a deep neural network model and identify the series category to which the cards belong; The identification strategy selection unit is used to select the appropriate identification strategy based on the classification results; The information extraction unit is used to extract card information according to the selected recognition strategy; The information integration unit is used to integrate classification results and identification information to determine the card type, name, and rarity information.

[0077] In some example embodiments of this disclosure, based on the foregoing scheme, the device further includes an anomaly handling module, used to initiate a recognition assistance request to the audience when the card recognition confidence level is lower than a set threshold; collect feedback information provided by the audience as training data; optimize the recognition model using user feedback data based on supervised learning; and record user evaluation data on the recognition results.

[0078] In some example embodiments of this disclosure, based on the foregoing scheme, the device further includes a system optimization module, used to deploy the recognition service using a distributed computing architecture; prioritize the recognition tasks; filter low-confidence samples through an active learning strategy; and introduce a misjudgment scenario construction technique.

[0079] In some example embodiments of this disclosure, based on the foregoing solution, the device further includes a multi-game adaptation module, used to provide a card template upload interface; establish independent recognition sub-models and databases for different card games; automatically select the corresponding recognition model and database based on live broadcast tag information; and quickly adapt to new card types through transfer learning.

[0080] In some example embodiments of this disclosure, based on the foregoing solution, the device further includes a database management module, used to collect basic card information through web crawler technology; regularly update the market valuation and popularity data of the cards; establish independent database subsets for different card games; and automatically switch the corresponding database subset based on the tag information of the live broadcast content.

[0081] The aforementioned device enables enhanced live streaming interaction based on visual recognition. It automatically identifies card information, combines it with bullet screen analysis for comprehensive scoring, and generates real-time visual feedback, thereby improving the interactivity and professionalism of the live stream. The modular design of this device allows for independent development and optimization of each functional module, improving the system's maintainability and scalability.

[0082] It is understood that the modules mentioned above refer to computer programs or program segments used to perform one or more specific functions. Furthermore, the distinction between these modules does not imply that the actual program code must also be separate.

[0083] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will be able to make various obvious changes, readjustments, and substitutions without departing from the scope of protection of the present invention. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for enhancing live streaming interaction based on visual recognition, characterized in that, include: In response to the triggering of card display events during live streaming, card images from the live video stream are acquired in real time. This real-time acquisition includes: obtaining real-time video data through the video stream interface provided by the live streaming platform; performing frame analysis on the real-time video data using a target detection algorithm, the frame analysis including: processing video frames using a target detection model, detecting card targets in the video frames using the target detection model, outputting the coordinate information of the card area, extracting the region of interest (ROI) based on the coordinate information, preprocessing the ROI including size normalization and color space conversion; performing image enhancement processing on the detected card area, the image enhancement processing including multi-frame fusion and edge optimization; and outputting the processed card image as the image to be recognized. The card images are processed to obtain card information, including card type, name, and rarity. The processing includes: classifying the card images using a deep neural network model to identify the card's category; selecting an appropriate recognition strategy based on the classification results; extracting key information using optical character recognition (OCR) technology for cards with clear text; comparing cards with distinct features using an image feature matching algorithm; and determining the card's type, name, and rarity by combining the classification results and recognition information. The deep neural network model uses a convolutional neural network architecture and is fine-tuned using card image data based on a pre-trained model. The OCR engine is used for optical character recognition, combined with post-processing logic to remove background interference. The image feature matching algorithm uses either a scale-invariant feature transform algorithm or a fast rotation-invariant feature transform algorithm to address recognition issues caused by card rotation, scaling, and partial occlusion. A confidence scoring mechanism is used to select the best matching result. The identified card information is matched and queried with a preset card database to obtain the corresponding basic value data, rarity data, and popularity index data; wherein, the popularity index is calculated based on the normalized monthly search volume or platform mentions. Within a preset time window for card display, bullet screen information during the live stream is collected, keywords in the bullet screen are extracted, and the sentiment tendency and frequency of occurrence of the keywords are analyzed; wherein, a dictionary containing sentiment keywords and evaluative keywords is pre-established, and each keyword is preset with a sentiment value and weight factor; Based on the aforementioned basic value data, rarity data, popularity index data, and bullet screen analysis results, the card's overall score is calculated using a preset comprehensive scoring model. The bullet screen score is obtained by summing the products of the weight factors and frequency of occurrence of each keyword within a statistical time window, and then dividing by the total number of bullet screens within that time window. The calculation formula for the comprehensive scoring model is: Comprehensive Score = α × Normalized Basic Value + β × Rarity Value + γ × Popularity Index + θ × Bullet Screen Score, where α, β, γ, and θ are preset weight coefficients, and the normalized basic value is calculated by logarithmic normalization of the basic value. Based on the comprehensive score, corresponding visual feedback content is generated and superimposed onto the live stream screen in real time for display. The generation of corresponding visual feedback content includes: determining the feedback level based on the comprehensive score, with different feedback levels corresponding to different visual effects; generating visual elements using a graphics rendering engine, including star ratings, light effect animations, and text information; and superimposing the visual elements onto the live stream output screen through the plugin interface of the live streaming software.

2. The live streaming interaction enhancement method based on visual recognition according to claim 1, characterized in that, The method also includes exception handling and user participation steps, specifically including: When the confidence level of card recognition falls below a set threshold, a recognition assistance request is sent to the audience. Collect audience feedback as training data; Optimize the recognition model using user feedback data based on supervised learning; Record user evaluation data on the recognition results, including error correction information and rating voting results.

3. The live streaming interaction enhancement method based on visual recognition according to claim 1, characterized in that, The method also includes a system performance optimization step, specifically including: The identification service is deployed using a distributed computing architecture, and processing efficiency is improved through load balancing. Prioritize and schedule recognition tasks to keep the delay in real-time feedback within a set threshold. Low-confidence samples were manually labeled using an active learning strategy. By introducing a misjudgment scenario construction technique, the model's ability to distinguish similar cards is improved.

4. The live streaming interaction enhancement method based on visual recognition according to claim 1, characterized in that, The method also includes a multi-card game adaptation step, specifically including: Provides a card template upload interface, supporting the addition of new card types; Establish independent recognition sub-models and databases for different card games; Automatically select the corresponding recognition model and database based on live stream tag information; By using transfer learning, the pre-trained model is fine-tuned using new card data to adapt to new card types.

5. The live streaming interaction enhancement method based on visual recognition according to claim 1, characterized in that, The method also includes steps for building and updating the card database, specifically including: Basic card information was collected from multiple third-party card platforms using web crawling technology; Regularly update the market valuation and popularity data of the cards; Establish independent database subsets for different card games; Based on the tag information of the live broadcast content, the system automatically switches to the corresponding database subset for query matching.

6. A live interactive enhancement device based on visual recognition according to any one of the enhancement methods in claims 1-4, characterized in that, include: The image acquisition module is used to acquire card images from the live video stream in real time in response to the card display event during the live broadcast. Specifically, it acquires real-time video data through the video stream interface provided by the live broadcast platform, performs frame analysis on the real-time video data using a target detection algorithm, including processing video frames with a target detection model, detecting card targets in the video frames and outputting the coordinate information of the card area, extracting the region of interest (ROI) of the card based on the coordinate information, performing preprocessing on the ROI including size normalization and color space conversion, and performing image enhancement processing on the detected card area including multi-frame fusion and edge optimization. The image recognition module is used to recognize and process the card images to obtain the card type, name, and rarity information. Specifically, a deep neural network model is used to classify the card images, identify the series category to which the card belongs, select an appropriate recognition strategy based on the classification results, use optical character recognition technology to extract key information for cards with clear text information, and use an image feature matching algorithm for comparison for cards with obvious features. The classification results and recognition information are combined to determine the card type, name, and rarity information, and the best matching result is selected through a confidence scoring mechanism. The data matching module is used to match and query the identified card information with a preset card database to obtain the corresponding basic value data, rarity data, and popularity index data; wherein, the popularity index is calculated based on the normalized monthly search volume or the number of platform mentions. The bullet screen analysis module is used to collect bullet screen information during the live broadcast within a preset time window of card display, extract keywords from the bullet screen, and analyze the sentiment tendency and frequency of occurrence of the keywords; wherein, a dictionary containing sentiment keywords and evaluative keywords is pre-established, and each keyword is preset with sentiment value and weight factor; The comprehensive scoring module is used to calculate the comprehensive score of the card based on the basic value data, rarity data, popularity index data, and bullet screen analysis results, using a preset comprehensive scoring model. The calculation formula for the comprehensive scoring model is: Comprehensive Score = α × Normalized Basic Value + β × Rarity Value + γ × Popularity Index + θ × Bullet Screen Score, where α, β, γ, and θ are preset weight coefficients. The normalized basic value is obtained by logarithmic normalization of the basic value. The bullet screen score is obtained by summing the products of the weight factors and frequency of occurrence of each keyword within a statistical time window, and then dividing by the total number of bullet screens within that time window. The visualization feedback module is used to generate corresponding visual feedback content based on the comprehensive score, and to overlay the visual feedback content onto the live broadcast screen in real time for display. The feedback level is determined based on the comprehensive score, and different feedback levels correspond to different visual effects. A graphics rendering engine is used to generate visual elements including star ratings, light effect animations, and text information, and the visual elements are overlaid onto the live broadcast output screen through the plugin interface of the live streaming software.

Images