Image recognition method and system based on machine learning

Abstract

The invention relates to the technical field of artificial intelligence and social media public opinion monitoring, in particular to an image recognition method and system based on machine learning, and the method comprises the steps: carrying out the OCR text extraction and regional visual feature extraction of an input image, generating a text emotion embedding vector in combination with a preset brand emotion dictionary, and carrying out the recognition of the emotion embedding vector; and visual and text features are fused through a cross-modal adversarial encoder to obtain joint semantic embedding representation, and whether hidden negative public opinions are formed or not is judged through a brand deviation degree discriminator. According to the method, semantic antagonism image contents including derogatory characters, competitive product comparison or sensitive symbols and the like can be effectively identified, the detection rate of recessive negative public opinions is improved, and the method is suitable for real-time monitoring of a large-scale social platform.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer vision technology, specifically an image recognition invention and system based on machine learning. Background Technology

[0002] In existing social media public opinion management, user-posted images often contain subtle brand denigration, competitor comparisons, or sensitive symbols (such as blurred logos or satirical photoshopped images). When dealing with such content with "semantic adversarial" nature, general image recognition models usually rely solely on visual features for classification, lacking a deep understanding of the semantic relationship between textual information in images and brand meaning, resulting in some implicit negative public opinion not being effectively identified.

[0003] To improve recognition capabilities, existing methods attempt to combine OCR technology to extract text from images and match it with brand keywords. However, this approach often employs a strategy of processing visual and textual modalities independently, failing to model the adversarial combination relationship between the two within a unified semantic space. This is especially true when faced with deliberately disguised or semantically inverted image and text content, where recognition accuracy is limited. Therefore, existing technologies require further improvement and development. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide an image recognition method, device and system based on machine learning, which addresses the above-mentioned deficiencies of the prior art. It aims to solve the problem that general image recognition models in social media public opinion management cannot effectively detect "semantic adversarial" image content such as implicit brand disparagement, competitor comparison or sensitive symbols. In particular, existing image-text joint analysis methods lack cross-modal semantic alignment mechanisms, making it difficult to model the adversarial combination relationship between visual and text in the latent space, resulting in a high rate of underreporting of negative public opinion.

[0005] The technical solution adopted by this invention to solve the technical problem is as follows: This invention provides an image recognition method based on machine learning, comprising: Perform optical character recognition on the input image to extract the text content from the image; The input image is subjected to region feature extraction to obtain multiple local visual feature vectors; The text content is matched with a preset brand sentiment dictionary to generate a text sentiment embedding vector; The multiple local visual feature vectors and the text sentiment embedding vectors are input into a cross-modal adversarial encoder to obtain a joint semantic embedding representation; The joint semantic embedding representation is input into the brand deviation discriminator, which outputs a judgment result on whether the image constitutes implicit negative public opinion.

[0006] In some embodiments of this application, the step of performing optical character recognition on the input image and extracting the text content from the image includes: The input image is processed by a convolutional recurrent neural network based on CTC loss, and a character sequence is output. The character sequence is cleaned to remove meaningless symbols and noise characters, retaining keywords with semantic value; The cleaned keywords are bound to the coordinates of the input image according to their occurrence positions to form a set of text fragments with spatial location information.

[0007] In some embodiments of this application, the step of extracting region features from the input image to obtain multiple local visual feature vectors includes: The Faster R-CNN model is used to detect several candidate regions in the input image, and each candidate region corresponds to a bounding box; A 1024-dimensional feature vector is extracted from each image patch within the bounding box using a ResNet-50 backbone network; Each feature vector is concatenated with its corresponding bounding box coordinates to form a set of local visual feature vectors with spatial location information.

[0008] In some embodiments of this application, the step of matching the text content with a preset brand sentiment dictionary to generate a text sentiment embedding vector includes: The brand sentiment dictionary is constructed, which includes three categories of entries: positive brand keywords, negative antagonistic words, and competitor identifiers, and sentiment polarity weights are assigned to each category of entries; For each keyword in the text content of the text fragment set, perform exact matching or fuzzy matching in the brand sentiment dictionary. If the match is successful, generate an initial sentiment score based on its sentiment polarity weight. The initial sentiment scores of all the text content are weighted and averaged, and then mapped to the interval [-1,1] using the tanh function, which is used as the text sentiment embedding vector.

[0009] In some embodiments of this application, the step of inputting the plurality of local visual feature vectors and the text sentiment embedding vector into a cross-modal adversarial encoder to obtain a joint semantic embedding representation includes: Construct the cross-modal adversarial encoder; wherein, The cross-modal adversarial encoder includes a visual projection layer, a text projection layer, and a multi-head cross-attention layer; Each of the local visual feature vectors is mapped to a d-dimensional latent space through a visual projection layer to obtain a visual token sequence; The text sentiment embedding vector is expanded through the text projection layer into a text token sequence with the same number of visual tokens as the visual token sequence; The visual token sequence and the text token sequence are input into the multi-head cross-attention layer to calculate the interaction response of the visual token sequence and the text token sequence in the d-dimensional latent space, and output the fused joint semantic embedding representation.

[0010] In some embodiments of this application, the step of inputting the joint semantic embedding representation into a brand deviation discriminator and outputting a determination result on whether the image constitutes implicit negative public opinion includes: The brand deviation discriminant is constructed, which consists of a two-layer fully connected network and a Sigmoid activation function; The fused joint semantic embedding representation is input into the brand deviation discriminator, and a value k between 0 and 1 is output. If the value k is greater than a preset threshold τ, the image is determined to contain hidden negative public opinion content; otherwise, it is determined not to contain hidden negative public opinion content and is considered normal content. 0.65 < τ ≤ 0.85.

[0011] In some embodiments of this application, it also includes: Obtain the number of times X1 and X2 constitute hidden negative public opinion within 24 hours and 48 hours respectively, and calculate the difference p, p=X1-X2; When the difference p ≥ 0, the threshold τ is corrected, and the preset range of the threshold τ is increased, with the increased preset range being used as the preset threshold τ; wherein, The adjusted preset range is 0.7 < τ ≤ 0.9; When the difference p < 0, the threshold τ is corrected, and the preset range of the threshold τ is lowered, with the lowered preset range being used as the preset threshold τ; wherein, The adjusted preset range is 0.6 < τ ≤ 0.8.

[0012] In some embodiments of this application, it also includes: Obtain the number of times Y1 and Y2 within 24 hours and 48 hours respectively that the judgment result is considered normal content without hidden negative public opinion content, and calculate the difference i, i=Y1-Y2; When the difference i ≥ 0, the corrected threshold τ is further corrected, and the preset range of the threshold τ is lowered. The lowered preset range is then used as the preset threshold τ. The adjusted preset range is 0.8 < τ ≤ 1; When the difference i < 0, the corrected threshold τ is further corrected, and the preset range of the threshold τ is increased. The increased preset range is then used as the preset threshold τ. The adjusted preset range is 0.5 < τ ≤ 0.7.

[0013] Based on the same technical concept, the present invention also provides a machine learning-based image recognition system, applied to the aforementioned machine learning-based image recognition method, comprising: The text extraction module is used to perform optical character recognition on the input image and extract the text content from the image; The visual feature extraction module is used to extract region features from the input image to obtain multiple local visual feature vectors; The emotion embedding generation module is used to match the text content with a preset brand emotion dictionary to generate a text emotion embedding vector. A cross-modal coding module is used to input the local visual feature vector and the text sentiment embedding vector into a cross-modal adversarial encoder to obtain a joint semantic embedding representation; The public opinion judgment module is used to input the joint semantic embedding representation into the brand deviation discriminator and output the judgment result of whether the image constitutes implicit negative public opinion.

[0014] In some embodiments of this application, the text extraction module uses a convolutional recurrent neural network based on CTC loss to perform optical character recognition processing, and cleans the output character sequence to form a set of text fragments with spatial location information.

[0015] Beneficial Effects: Compared with existing technologies, this invention provides a machine learning-based image recognition method. By constructing a brand sentiment dictionary, it assigns sentiment semantic weights to the text extracted by OCR. Combined with local visual region features extracted by Faster R-CNN, it achieves deep fusion of visual and textual elements in a unified latent space within a cross-modal adversarial encoder. This enables the model to capture adversarial semantic patterns such as the co-occurrence of censored logos and sarcastic text, and combinations of competitor images and derogatory terms. The brand deviation discriminator directly quantifies the degree of deviation between the text / image content and the brand's positive image based on joint embedding. It does not rely on explicit keyword matching or independent modality classification, effectively improving the detection rate of implicit negative sentiment images. Simultaneously, it avoids strong dependence on costly manually labeled data, making it suitable for real-time sentiment monitoring scenarios on large-scale social media platforms. Attached Figure Description

[0016] Figure 1 This is a flowchart of the machine learning-based image recognition method in an embodiment of the present invention; Figure 2This is a functional block diagram of the machine learning-based image recognition system in an embodiment of the present invention. Detailed Implementation

[0017] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0018] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0019] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0020] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the communication between the inner sides of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0021] This invention provides an image recognition method and system based on machine learning. The specific embodiments of this invention will be described in detail below with reference to the accompanying drawings.

[0022] See Figure 1 As shown, an image recognition method based on machine learning includes: Perform optical character recognition on the input image to extract the text content from the image; Region feature extraction is performed on the input image to obtain multiple local visual feature vectors; The text content is matched with a preset brand sentiment dictionary to generate a text sentiment embedding vector; Multiple local visual feature vectors and text sentiment embedding vectors are input into a cross-modal adversarial encoder to obtain a joint semantic embedding representation; The joint semantic embedding representation is input into the brand deviation discriminator, which outputs the judgment result of whether the image constitutes implicit negative public opinion.

[0023] In one specific embodiment of this application, optical character recognition is performed on an input image to extract text content from the image, including: A convolutional recurrent neural network based on CTC loss is used to perform OCR processing on the input image and output a character sequence; The character sequence is cleaned to remove meaningless symbols and noisy characters, while retaining keywords with semantic value; The cleaned keywords are bound to the coordinates of the input image according to their location, forming a set of text fragments with spatial location information.

[0024] In one specific embodiment of this application, region feature extraction is performed on the input image to obtain multiple local visual feature vectors, including: The Faster R-CNN model is used to detect several candidate regions in the input image, and each candidate region corresponds to a bounding box; A 1024-dimensional feature vector is extracted from each image patch within the bounding box using a ResNet-50 backbone network; Each feature vector is concatenated with its corresponding bounding box coordinates to form a set of local visual feature vectors with spatial location information.

[0025] In one specific embodiment of this application, the text content is matched with a preset brand sentiment dictionary to generate a text sentiment embedding vector, including: Construct a brand sentiment dictionary, which includes three categories of entries: positive brand keywords, negative antagonistic words, and competitor identifiers, and assign sentiment polarity weights to each category of entries; For each text content keyword in the text fragment set, perform exact or fuzzy matching in the brand sentiment dictionary. If the match is successful, generate an initial sentiment score based on its sentiment polarity weight. The initial sentiment scores of all text content are weighted and averaged, and then mapped to the interval [-1,1] using the tanh function to serve as the text sentiment embedding vector.

[0026] In one specific embodiment of this application, multiple local visual feature vectors and text sentiment embedding vectors are input into a cross-modal adversarial encoder to obtain a joint semantic embedding representation, including: Construct a cross-modal adversarial encoder; whereby, The cross-modal adversarial encoder consists of a visual projection layer, a text projection layer, and a multi-head cross-attention layer; Each local visual feature vector is mapped to a d-dimensional latent space through a visual projection layer, resulting in a visual token sequence; The text sentiment embedding vector is expanded into a text token sequence with the same number of visual tokens as the visual token sequence through a text projection layer; The visual token sequence and the text token sequence are input into a multi-head cross-attention layer. The interaction response of the visual token sequence and the text token sequence in the d-dimensional latent space is calculated, and the fused joint semantic embedding representation is output.

[0027] In one specific embodiment of this application, the joint semantic embedding representation is input into a brand deviation discriminator, and the result of determining whether the image constitutes implicit negative public opinion is output, including: A brand deviation discriminant is constructed, which consists of two fully connected layers and a sigmoid activation function; The fused joint semantic embedding representation is input into the brand deviation discriminator, and a value k between 0 and 1 is output. If the value k is greater than the preset threshold τ, the image is determined to contain hidden negative public opinion content; otherwise, it is determined not to contain hidden negative public opinion content and is considered normal content. 0.65 < τ ≤ 0.85.

[0028] In one specific embodiment of this application, it further includes: Obtain the number of times X1 and X2 constitute hidden negative public opinion within 24 hours and 48 hours respectively, and calculate the difference p, p=X1-X2; When the difference p ≥ 0, the threshold τ is corrected, and the preset range of the threshold τ is increased. The increased preset range is then used as the preset threshold τ. The adjusted preset range is 0.7 < τ ≤ 0.9; When the difference p < 0, the threshold τ is corrected, and the preset range of the threshold τ is lowered. The lowered preset range is then used as the preset threshold τ. The adjusted preset range is 0.6 < τ ≤ 0.8.

[0029] In one specific embodiment of this application, it further includes: Obtain the number of times Y1 and Y2 within 24 hours and 48 hours respectively that the judgment result is considered normal content without hidden negative public opinion content, and calculate the difference i, i=Y1-Y2; When the difference i ≥ 0, the corrected threshold τ is further corrected, and the preset range of threshold τ is lowered. The lowered preset range is then used as the preset threshold τ. The adjusted preset range is 0.8 < τ ≤ 1; When the difference i < 0, the corrected threshold τ is further corrected, and the preset range of threshold τ is increased. The increased preset range is then used as the preset threshold τ. The adjusted preset range is 0.5 < τ ≤ 0.7.

[0030] See Figure 2 As shown, the present invention also provides a machine learning-based image recognition system, applied in machine learning-based image recognition methods, comprising: The text extraction module is used to perform optical character recognition on the input image and extract the text content from the image; The visual feature extraction module is used to extract region features from the input image and obtain multiple local visual feature vectors. The sentiment embedding generation module is used to match text content with a preset brand sentiment dictionary to generate text sentiment embedding vectors. The cross-modal coding module is used to input local visual feature vectors and text sentiment embedding vectors into the cross-modal adversarial encoder to obtain a joint semantic embedding representation; The public opinion discrimination module is used to input the joint semantic embedding representation into the brand deviation discrimination unit and output the judgment result of whether the image constitutes implicit negative public opinion.

[0031] In one specific embodiment of this application, the text extraction module uses a convolutional recurrent neural network based on CTC loss to perform optical character recognition processing, and cleans the output character sequence to form a set of text fragments with spatial location information.

[0032] In one specific embodiment of this application, the input image is fed into a text extraction module. This module employs a convolutional recurrent neural network structure based on the CTC loss function, specifically consisting of a 7-layer convolutional backbone followed by bidirectional LSTM layers. After passing through this network, the input image outputs a character sequence, representing all detected text content in the image. Subsequently, the character sequence undergoes cleaning processing, including removing consecutive repeating characters, non-ASCII symbols, and isolated character fragments with a length less than 2, while retaining semantically valuable keywords, such as "fake goods," "knockoffs," and "not as good as XX brand." The cleaned keywords are recorded according to their spatial location information in the original image, forming a set of text fragments with coordinate binding. Each text fragment contains the keyword content and its coordinates (x1, y1, x2, y2) of the top-left and bottom-right corners of the image. This set of text fragments serves as the input to the subsequent brand sentiment dictionary matching module 3.

[0033] Simultaneously, the input image is also fed into the visual feature extraction module. This module employs the Faster R-CNN object detection architecture, where the Region Proposal Network (RPN) generates several candidate regions on the input image, each corresponding to a bounding box. For each image patch enclosed by a bounding box, high-dimensional visual features are extracted using a ResNet-50 backbone network, outputting a 1024-dimensional local visual feature vector. To preserve spatial context information, each 1024-dimensional feature vector is concatenated with its corresponding bounding box coordinates (x1, y1, x2, y2) to form a 1028-dimensional local visual feature vector with spatial location information. The local visual feature vectors corresponding to all candidate regions form a set of local visual feature vectors, which serves as the input to the visual projection layer in the cross-modal adversarial encoder.

[0034] The sentiment embedding generation module receives a set of coordinated text fragments output by the text extraction module. This module has a pre-built brand sentiment dictionary containing three categories of terms: positive brand keywords (e.g., "authentic" "officially authorized"), negative adversarial terms (e.g., "counterfeit," "inferior," "failure"), and competitor identifiers (e.g., registered trademarks or common abbreviations of other brands). Each category is assigned a corresponding sentiment polarity weight: positive keywords have a weight of +1.0, negative adversarial terms have a weight of -1.0, and competitor identifiers have a weight of -0.6. For each keyword in a text fragment, the sentiment embedding generation module first performs an exact match in the brand sentiment dictionary; if no match is found, a fuzzy matching mechanism based on edit distance is initiated, with a successful match considered when the edit distance is less than or equal to 2. After a successful match, the corresponding sentiment polarity weight is assigned according to the term type, forming the initial sentiment score for that text fragment. The sentiment scores of all text fragments are weighted and averaged according to their frequency of occurrence in the image, then mapped to the [-1,1] interval using the tanh function, ultimately generating a scalar value, i.e., the text sentiment embedding vector. This vector serves as the input to the text projection layer in the cross-modal adversarial encoder.

[0035] The cross-modal adversarial encoder comprises local visual feature vector input, text sentiment embedding vector input, a visual projection layer, a text projection layer, a multi-head cross-attention layer, and a joint semantic embedding representation output. The local visual feature vector input receives a set of local visual feature vectors from the visual feature extraction module, each vector having a dimension of 1028. The visual projection layer is a fully connected layer with an input dimension of 1028 and an output dimension of d (d=512), used to map each local visual feature vector to a unified d-dimensional latent space, generating a visual token sequence V = {v1, v2, ..., v...}. n}, where n is the number of candidate regions. The text sentiment embedding vector input receives a scalar text sentiment embedding vector s∈[-1,1] from the output of the sentiment embedding generation module. The text projection layer consists of a learnable parameter matrix. and bias terms The structure expands the scalar s to a sequence of text tokens T = {t1, t2, ..., t}, which is the same as the number of visual tokens n. i}, where each t i =W t ·s+b t Subsequently, the visual token sequence V and the text token sequence T are simultaneously input into a multi-head cross-attention layer. This layer contains h=8 parallel attention heads, each of which computes the query, key, and value matrices.

[0036] Attention weight A k : ; The output is O k =A k V k The outputs of all heads are concatenated and then subjected to a linear transformation to obtain the fused joint semantic embedding representation. This representation captures the cross-modal interaction between each local visual region and the global textual sentiment, and serves as input to the brand deviation discriminator.

[0037] The brand deviation discriminator consists of two fully connected layers. The first layer has an input dimension of n×d, which is compressed into a d-dimensional vector through global average pooling, then activated by ReLU before being input into the second fully connected layer, with an output dimension of 1. The second layer is followed by a Sigmoid activation function, compressing the output value to the (0,1) interval. This output value represents the probability that the input image constitutes implicit negative public opinion. The system presets a threshold τ, with a value range of 0.65 < τ ≤ 0.85, preferably τ = 0.75. When the discriminator's output value is greater than τ, the image is determined to contain implicit negative public opinion content; otherwise, it is determined to be normal content.

[0038] To prevent the system from frequently identifying an image (or images) as hidden negative public opinion content, the judgment criteria (i.e., threshold) should be adjusted to control false alarms or missed alarms. The number of judgment results constituting hidden negative public opinion within 24 hours and 48 hours respectively are obtained as X1 and X2, and the difference p is calculated, p=X1-X2. When the difference p ≥ 0, the threshold τ is corrected, and the preset range of the threshold τ is increased. The increased preset range is then used as the preset threshold τ. The adjusted preset range is 0.7 < τ ≤ 0.9; the preferred value is τ = 0.8. When the difference p < 0, the threshold τ is corrected, and the preset range of the threshold τ is lowered. The lowered preset range is then used as the preset threshold τ. The preset range after the adjustment is 0.6 < τ ≤ 0.8; the preferred value is τ = 0.7.

[0039] In one specific embodiment of this application, it further includes: Obtain the number of times Y1 and Y2 within 24 hours and 48 hours respectively that the judgment result is considered normal content without hidden negative public opinion content, and calculate the difference i, i=Y1-Y2; When the difference i ≥ 0, the corrected threshold τ is further corrected, and the preset range of threshold τ is lowered. The lowered preset range is then used as the preset threshold τ. The preset range after adjustment is 0.8 < τ ≤ 1; the preferred value is τ = 0.9. When the difference i < 0, the corrected threshold τ is further corrected, and the preset range of threshold τ is increased. The increased preset range is then used as the preset threshold τ. The adjusted preset range is 0.5 < τ ≤ 0.7; the preferred value is τ = 0.6.

[0040] In practical deployment, the above modules are integrated into a smart terminal device, which includes a memory, a processor, and an image recognition program stored in the memory and running on the processor. When the processor executes the program, it sequentially calls the text extraction module, visual feature extraction module, sentiment embedding generation module, cross-modal coding module, and sentiment discrimination module to complete the entire process from raw image input to sentiment discrimination result output. The brand sentiment dictionary is stored in the memory and supports dynamic updates, such as adding recently frequently occurring derogatory terms or emerging competitor identifiers through manual review. The parameters of the Faster R-CNN model and the ResNet-50 backbone network are fine-tuned end-to-end during the training phase using a large-scale labeled dataset, where the labeled data includes images and their binary labels indicating whether they constitute implicit negative sentiment. During training, the cross-modal adversarial encoder and the brand deviation discriminator are jointly optimized, and the loss function adopts binary cross-entropy loss, with backpropagation updating all learnable parameters.

[0041] In real-world applications on social media platforms, the entire system can be deployed on a content moderation server cluster to batch scan user-uploaded images. When an image is identified as containing implicit negative sentiment, the system automatically marks it and pushes it to a manual review queue, while simultaneously recording the event's timestamp, user ID, and image hash value for subsequent sentiment trend analysis. Because this solution does not rely on explicit keyword triggers but instead captures adversarial intent within image-text combinations through cross-modal semantic modeling, it can effectively identify complex implicit disparaging patterns such as "a blurred logo accompanied by the text 'Is this really from XX?'" or "a competitor's product image overlaid with the text 'Crushing YY Brand'." The parameter configurations, threshold settings, and dictionary content for each module can be customized to meet specific brand monitoring needs, ensuring the system's adaptability and accuracy across different business scenarios.

[0042] To enable those skilled in the art to fully understand and implement this invention, the specific implementation principles of this invention are further supplemented below with a specific application scenario.

[0043] In the content moderation system of social media platforms, when a user uploads an image containing a product picture and overlaid text, the image is first sent to the text extraction module. For example, the image contains a handwritten text "Is this really from XX?", where "XX" is the name of a well-known brand, and its brand logo is partially blurred in the image. The text extraction module performs OCR processing on the image based on a convolutional recurrent neural network driven by the CTC loss function, outputting the original character sequence; then, it removes noise characters without semantic value through a cleaning strategy, retaining the keyword fragment "XX's?", and recording its spatial coordinates (x1, y1, x2, y2) in the image, forming a set of text fragments with coordinates, which serves as the basic input for subsequent processing.

[0044] The image is then fed into the visual feature extraction module. This module uses the Region Proposal Network (RPN) in the Faster R-CNN architecture to generate multiple candidate regions in the image. One candidate region exactly covers the masked brand logo area, and another covers the main product area. Each candidate region's corresponding image patch is processed by the ResNet-50 backbone network to extract a 1024-dimensional visual feature vector, which is then concatenated with its bounding box coordinates to form a 1028-dimensional local visual feature vector. This set of local visual feature vectors is then passed to the local visual feature vector input in the cross-modal adversarial encoder.

[0045] The sentiment embedding generation module receives a set of text fragments output from the text extraction module and matches them against a pre-built brand sentiment dictionary. This dictionary already includes "XX" as a positive brand keyword (weight +1.0). While the interrogative sentence "This is really...?" is not directly listed, it contains the brand name "XX" and the modal particle "really," prompting the system to initiate a fuzzy matching mechanism. The edit distance between this text fragment and the negative adversarial template "This is really YY's?" in the dictionary is calculated to be 1, which is less than the threshold of 2. Therefore, a successful match is determined, and the text fragment is assigned a sentiment polarity weight of -1.0. The sentiment scores of all text fragments are then averaged by frequency and compressed using the tanh function, outputting a scalar text sentiment embedding vector s = -0.82, which is used as input to the cross-modal adversarial encoder.

[0046] In the cross-modal adversarial encoder, the 1028-dimensional vector received by the local visual feature vector input 10 is mapped to a 512-dimensional visual token sequence V via the visual projection layer 12; the scalar s received by the text sentiment embedding vector input 11 is processed by the learnable parameter matrix W via the text projection layer 13. t and bias b t The sequence is expanded into a text token sequence T of the same length as V. V and T are then fed together into a multi-head cross-attention module. In this module, each attention head calculates the response strength of the visual region to global text sentiment, using V as the query, T as the key, and T as the value. For example, a visual token v covering a blurred logo... i A high response value obtained in the attention weight calculation indicates a strong correlation between the region and the negative text semantics. The outputs of the eight attention heads are concatenated and linearly transformed to generate a joint semantic embedding representation with a dimension of n×512, which fully encodes the adversarial interaction information between each visual region and the text sentiment.

[0047] The brand deviation discriminator receives the joint semantic embedding representation, first compresses it into a 512-dimensional vector through global average pooling, then activates it with ReLU before inputting it into a second fully connected network, and finally outputs a probability value of 0.78 through the Sigmoid function. Since this value is greater than the preset threshold τ=0.75, the system determines that the image constitutes implicit negative public opinion content. This determination does not rely on the appearance of explicit derogatory words such as "fake goods" or "knock-offs," but rather stems from the cross-modal adversarial encoder's collaborative modeling of the visually anomalous region of "redacted logo" and the ironic text "Is this really from XX?" in the semantic latent space, thus achieving accurate capture of semantically adversarial content.

[0048] In the above processing flow, the text extraction module and the visual feature extraction module run in parallel, outputting a set of text fragments and a set of local visual feature vectors, respectively; the sentiment embedding generation module starts after OCR is completed, generating text sentiment embedding vectors; the cross-modal adversarial encoder synchronously receives the outputs from the visual feature extraction module and the sentiment embedding generation module, completing image-text alignment and interaction in a unified d-dimensional space; the brand deviation discriminator makes the final judgment based on the fused semantic representation. The entire process, through tensor-level data transfer and end-to-end training mechanisms, ensures that the system can still stably output high-confidence negative public opinion judgment results when facing complex implicit denigration patterns such as "competitor product images superimposed with the text 'crushing YY brand'" or "authentic product packaging labeled with 'failure record'".

[0049] The above description is merely one embodiment of the present invention, but it cannot be used to limit the scope of the present invention. Any structural changes made based on the present invention, as long as they do not lose the essence of the present invention, should be considered to fall within the protection scope of the present invention and be subject to its restrictions.

[0050] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the system described above can be found in the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0051] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.

[0052] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.

[0053] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.

[0054] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

[0055] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. An image recognition method based on machine learning, characterized in that, include: Perform optical character recognition on the input image to extract the text content from the image; The input image is subjected to region feature extraction to obtain multiple local visual feature vectors; The text content is matched with a preset brand sentiment dictionary to generate a text sentiment embedding vector; The multiple local visual feature vectors and the text sentiment embedding vectors are input into a cross-modal adversarial encoder to obtain a joint semantic embedding representation; The joint semantic embedding representation is input into the brand deviation discriminator, which outputs a judgment result on whether the image constitutes implicit negative public opinion.

2. The image recognition method based on machine learning according to claim 1, characterized in that, The step of performing optical character recognition on the input image and extracting the text content from the image includes: The input image is processed by a convolutional recurrent neural network based on CTC loss, and a character sequence is output. The character sequence is cleaned to remove meaningless symbols and noise characters, retaining keywords with semantic value; The cleaned keywords are bound to the coordinates of the input image according to their occurrence positions to form a set of text fragments with spatial location information.

3. The image recognition method based on machine learning according to claim 2, characterized in that, The step of extracting region features from the input image to obtain multiple local visual feature vectors includes: The Faster R-CNN model is used to detect several candidate regions in the input image, and each candidate region corresponds to a bounding box; A 1024-dimensional feature vector is extracted from each image patch within the bounding box using a ResNet-50 backbone network; Each feature vector is concatenated with its corresponding bounding box coordinates to form a set of local visual feature vectors with spatial location information.

4. The image recognition method based on machine learning according to claim 3, characterized in that, The step of matching the text content with a preset brand sentiment dictionary to generate a text sentiment embedding vector includes: The brand sentiment dictionary is constructed, which includes three categories of entries: positive brand keywords, negative antagonistic words, and competitor identifiers, and sentiment polarity weights are assigned to each category of entries; For each keyword in the text content of the text fragment set, perform exact matching or fuzzy matching in the brand sentiment dictionary. If the match is successful, generate an initial sentiment score based on its sentiment polarity weight. The initial sentiment scores of all the text content are weighted and averaged, and then mapped to the interval [-1,1] using the tanh function, which is used as the text sentiment embedding vector.

5. The image recognition method based on machine learning according to claim 4, characterized in that, The step of inputting multiple local visual feature vectors and the text sentiment embedding vector into a cross-modal adversarial encoder to obtain a joint semantic embedding representation includes: Construct the cross-modal adversarial encoder; wherein, The cross-modal adversarial encoder includes a visual projection layer, a text projection layer, and a multi-head cross-attention layer; Each of the local visual feature vectors is mapped to a d-dimensional latent space through a visual projection layer to obtain a visual token sequence; The text sentiment embedding vector is expanded through the text projection layer into a text token sequence with the same number of visual tokens as the visual token sequence; The visual token sequence and the text token sequence are input into the multi-head cross-attention layer to calculate the interaction response of the visual token sequence and the text token sequence in the d-dimensional latent space, and output the fused joint semantic embedding representation.

6. The image recognition method based on machine learning according to claim 5, characterized in that, The step of inputting the joint semantic embedding representation into the brand deviation discriminator and outputting a determination result on whether the image constitutes implicit negative public opinion includes: The brand deviation discriminant is constructed, which consists of a two-layer fully connected network and a Sigmoid activation function; The fused joint semantic embedding representation is input into the brand deviation discriminator, and a value k between 0 and 1 is output. If the value k is greater than a preset threshold τ, the image is determined to contain hidden negative public opinion content; otherwise, it is determined not to contain hidden negative public opinion content and is considered normal content. 0.65 < τ ≤ 0.

85.

7. The image recognition method based on machine learning according to claim 6, characterized in that, Also includes: Obtain the number of times X1 and X2 constitute hidden negative public opinion within 24 hours and 48 hours respectively, and calculate the difference p, p=X1-X2; When the difference p ≥ 0, the threshold τ is corrected, and the preset range of the threshold τ is increased, with the increased preset range being used as the preset threshold τ; wherein, The adjusted preset range is 0.7 < τ ≤ 0.9; When the difference p < 0, the threshold τ is corrected, and the preset range of the threshold τ is lowered, with the lowered preset range being used as the preset threshold τ; wherein, The adjusted preset range is 0.6 < τ ≤ 0.

8.

8. The image recognition method based on machine learning according to claim 7, characterized in that, Also includes: Obtain the number of times Y1 and Y2 within 24 hours and 48 hours respectively that the judgment result is considered normal content without hidden negative public opinion content, and calculate the difference i, i=Y1-Y2; When the difference i ≥ 0, the corrected threshold τ is further corrected, and the preset range of the threshold τ is lowered. The lowered preset range is then used as the preset threshold τ. The adjusted preset range is 0.8 < τ ≤ 1; When the difference i < 0, the corrected threshold τ is further corrected, and the preset range of the threshold τ is increased. The increased preset range is then used as the preset threshold τ. The adjusted preset range is 0.5 < τ ≤ 0.

7.

9. A machine learning-based image recognition system, applied in the machine learning-based image recognition method as described in any one of claims 1-8, characterized in that, include: The text extraction module is used to perform optical character recognition on the input image and extract the text content from the image; The visual feature extraction module is used to extract region features from the input image to obtain multiple local visual feature vectors; The emotion embedding generation module is used to match the text content with a preset brand emotion dictionary to generate a text emotion embedding vector. A cross-modal coding module is used to input the local visual feature vector and the text sentiment embedding vector into a cross-modal adversarial encoder to obtain a joint semantic embedding representation; The public opinion judgment module is used to input the joint semantic embedding representation into the brand deviation discriminator and output the judgment result of whether the image constitutes implicit negative public opinion.

10. The image recognition system based on machine learning according to claim 9, characterized in that, The text extraction module uses a convolutional recurrent neural network based on CTC loss to perform optical character recognition processing, and cleans the output character sequence to form a set of text fragments with spatial location information.

Images