A method for identifying decommissioned photovoltaic module labels

Abstract

This invention relates to the field of image processing technology, and more particularly to a method for identifying labels on decommissioned photovoltaic modules. The method includes: acquiring images of decommissioned photovoltaic modules; extracting text characters and spatial location information from the nameplate area; performing semantic modeling on the text characters to obtain text semantic features; modeling the text spatial location to obtain text layout structure features; constructing visual appearance features and visual layout structure features using the nameplate area image; inputting multimodal features into a recognition model; and outputting key information. This invention addresses the problem in existing decommissioned photovoltaic module nameplate information recognition processes where, due to the complex layout, irregular field arrangement, and diverse text expression forms of nameplate images, recognition methods relying solely on single text or visual features struggle to accurately understand the semantic structure of the nameplate, resulting in low accuracy in identifying key information.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for identifying tags on decommissioned photovoltaic modules. Background Technology

[0002] Retired photovoltaic modules need to be processed in accordance with relevant recycling and resource recycling requirements. At the same time, the photovoltaic module full life cycle responsibility management mechanism should be followed to effectively trace and manage the flow and use information of the modules in each stage of production, sales, service, retirement and dismantling.

[0003] Existing technologies still have the following shortcomings when identifying and extracting nameplate information from retired photovoltaic modules: 1. Photovoltaic modules produced by different manufacturers have significant differences in nameplate design, and their information content organization and layout format are inconsistent. When expanding or associating the detected keyword text area, text area concatenation or mismatch may occur, which will affect the accurate recognition of key information by the character recognition model.

[0004] 2. Photovoltaic modules produced by different manufacturers, or modules from different production batches of the same manufacturer, often use different text descriptions for the same key information fields on their nameplates; at the same time, there are significant differences in the layout structure, expression of key information keywords, font type and font size of the nameplates.

[0005] According to CN120014619A, when the nameplate of a retired photovoltaic module has multiple columns, field names and field values ​​are not on the same line, parameter areas are arranged irregularly, the nameplate layouts of different manufacturers are significantly different, or the same field has multiple Chinese, English and abbreviation expressions, this identification method based on single text features or rule matching is prone to problems such as field confusion, missed detection or misjudgment. Summary of the Invention

[0006] To address the shortcomings of existing methods, this invention addresses the problem that in the process of identifying nameplate information of retired photovoltaic modules, the complex layout of the nameplate images, irregular field arrangement, and diverse text expression forms make it difficult for identification methods relying solely on single text or visual features to accurately understand the semantic structure of the nameplate, resulting in a low accuracy rate for identifying key information.

[0007] The technical solution adopted in this invention is: a method for identifying tags on decommissioned photovoltaic modules, comprising the following steps: Step 1: Collect images of retired photovoltaic modules and extract text characters and spatial location information from the nameplate area; As a preferred embodiment of the present invention, image cropping, orthodontic processing and angle correction preprocessing are performed.

[0008] In a preferred embodiment of the present invention, the spatial location information includes: boundary coordinates, region width, and region height.

[0009] In a preferred embodiment of the present invention, the text characters include: component weight, component size, maximum power voltage, maximum power current, and output power.

[0010] Step 2: Perform semantic modeling on the text characters to obtain text semantic features; model the text spatial location to obtain text layout structure features; construct visual appearance features and visual layout structure features using the nameplate area image; input the multimodal features into the recognition model and output key information; In a preferred embodiment of the present invention, step two specifically includes: Step 21: Divide the text content into several tokens by word segmentation or character-level segmentation, and divide the nameplate image into several image blocks; Step 22: Process the token using a text embedding layer to generate a text semantic embedding vector. ; Step 23: Process the text region positions of the token and image block using two-dimensional positional encoding to generate 2D text positional embedding vectors respectively. and 2D image patch location embedding vector ; Step 24: Process the text region positions of the token and image patch using one-dimensional sequence position encoding to generate a 1D text sequence embedding vector. and 1D image patch sequential embedding vector ; Step 25: Vectorize and expand the image patch, input it into the linear embedding mapping layer, and generate visual patch embedding vectors. ; As a preferred embodiment of the present invention The formula is: ; in, Represents a linear mapping matrix; Indicates the bias term. For the first j Image blocks, : No. j The one-dimensional pixel vector formed by unfolding an image patch.

[0011] Step 26, place the first i Text token fusion ; Step 27, place the first j Image patch fusion ; Step 28, and By concatenating the features, multimodal features are obtained. .

[0012] In a preferred embodiment of the present invention, the recognition model includes a Transformer network.

[0013] As a preferred embodiment of the present invention, a system for identifying labels on decommissioned photovoltaic modules includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement a method for identifying labels on decommissioned photovoltaic modules.

[0014] In a preferred embodiment of the present invention, a computer-readable medium storing computer program code implements a method for identifying tags on decommissioned photovoltaic modules when executed by a processor.

[0015] The beneficial effects of this invention are: 1. This invention introduces text semantic features, layout structure features and visual appearance features simultaneously in the process of nameplate information recognition, and performs joint modeling of information from different modalities. This enables the model to not only understand the semantic meaning of the text itself, but also to comprehensively consider the spatial layout relationship of the text in the nameplate and the corresponding visual expression, thereby significantly improving the overall understanding of the complex layout and diverse typesetting of retired photovoltaic module nameplates. 2. This invention avoids the misjudgment problems that are easily caused by relying solely on single text recognition results or manual rule matching methods in traditional technologies. Through collaborative modeling of multimodal features, the system can stably and accurately identify key information fields in nameplates in real-world application scenarios with mixed multilingual text, multiple expressions of synonymous fields, and differences in nameplate design, thereby improving the reliability of the recognition results. 3. The method of this invention is applicable to retired photovoltaic module nameplates from different manufacturers, models, and structural forms. Even when faced with complex situations such as changes in nameplate layout, differences in field expression, and inconsistent image quality, it can still maintain a high recognition accuracy and has good generalization ability. At the same time, this method does not require rule customization for specific manufacturers or specific nameplate structures, and has strong engineering practical value and prospects for promotion and application. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method for identifying tags on decommissioned photovoltaic modules according to the present invention; Figure 2 These are label images from different manufacturers or batches; Figure 3 These are comparison images before and after image preprocessing; Figure 4 This is an image showing the OCR recognition result; Figure 5 This invention relates to an information annotation method; Figure 6 This is a schematic diagram of the multimodal fusion mechanism of the present invention; Figure 7 This is a schematic diagram illustrating the effect of nameplate information recognition. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0018] To address the issues of field confusion, missed detections, or misjudgments in existing methods, this invention proposes a method for recognizing nameplate information of decommissioned photovoltaic modules that simultaneously combines textual semantic information, spatial layout structure information, and image visual appearance information. This enables the recognition model not only to identify the text characters themselves but also to understand the spatial relationships of the text within the nameplate, the correspondence between field names and field values, and the overall visual structural features of the nameplate. This improves the accuracy of key information field recognition in scenarios with complex layouts, multiple manufacturers, multiple languages, and multiple expressions. Compared to existing methods, this invention introduces textual semantic features, two-dimensional spatial location features, visual appearance features, and visual layout structure features, and utilizes a multimodal Transformer network to jointly model different modalities of information. This allows for the establishment of associations between field names, field values, field units, and their respective image regions within a unified feature space. This approach avoids the limitations of relying solely on keyword line expansion, manual rule matching, or local OCR post-processing, enabling the model to understand the overall structural semantics of decommissioned photovoltaic module nameplates. This enhances its adaptability to scenarios with complex layouts, diverse field expressions, and inconsistent image quality.

[0019] like Figure 1 As shown, a method for identifying tags on decommissioned photovoltaic modules includes the following steps: Step 1: Collect images of decommissioned photovoltaic modules and extract text characters from the nameplate area in the module images to obtain the text characters contained in the nameplate and their corresponding spatial location information, and construct a data sample containing text characters and their spatial locations. First, physical samples of retired photovoltaic modules are collected by visiting abandoned photovoltaic module recycling stations, storage warehouses, and installation sites. Images of the nameplates of retired photovoltaic modules are then obtained using industrial cameras, mobile terminals, or on-site data collection equipment. The nameplates cover photovoltaic modules from different manufacturers, models, and years of use to ensure the diversity and representativeness of the data sample.

[0020] Collect photovoltaic modules of various brands and models, such as Figure 2As shown, the dataset includes well-known brands and regional brands to ensure access to different types of labels; to enhance the diversity of the dataset, the image samples include various photovoltaic components and their labels to ensure the generalization ability of the model.

[0021] Data preprocessing is performed on the collected images of nameplates of retired photovoltaic modules to improve the stability and accuracy of subsequent text detection, recognition, and multimodal feature extraction.

[0022] Data preprocessing includes: image cropping, orthogonalization, and angle correction.

[0023] Among them, image cropping is used to remove background areas that are not related to the nameplate information and retain the effective image area containing the main information of the nameplate, thereby reducing the impact of background noise on subsequent model processing; Orthogonal processing is used to correct the distortion of nameplate images caused by differences in shooting angle, lens distortion, or installation position, so that the nameplate area appears as a regular, near-planar structure in the image; Angle correction processing is used to rotate and correct tilted nameplate images, making the text information in the nameplate more horizontal or vertical, thereby enhancing the readability and structural consistency of the text area.

[0024] The above data preprocessing operations ensure that the input image maintains consistency in geometric structure and orientation, providing high-quality input data for subsequent OCR processing, multimodal feature construction, and model training. Figure 3 This is a comparison chart showing the data before and after preprocessing.

[0025] Optical Character Recognition (OCR) is performed on the nameplate image after image preprocessing. By combining character detection and character recognition, the text areas in the nameplate image are automatically located, and the character content in each text area is recognized, thereby obtaining the text information contained in the nameplate.

[0026] like Figure 4 As shown, while performing OCR processing, the spatial position information of each text region in the nameplate image is extracted and recorded simultaneously. The spatial position information is used to characterize the layout distribution features of the text in the nameplate image, including at least the boundary coordinate information, region width, and region height of the corresponding text region. The boundary coordinates can be represented by the coordinates of the upper left and lower right corners of the rectangle or the coordinates of the center point and the size information.

[0027] The text recognition results obtained by OCR processing are associated one by one with their corresponding spatial location information and stored in a unified structured data form to form an intermediate dataset containing text semantic information and its spatial layout features. The intermediate data provides basic data support for subsequent layout understanding, key information extraction and structured parsing.

[0028] As Figure 5 shown, the marked content includes the key information fields that need to be extracted重点 from the nameplates of retired photovoltaic modules. The key information fields are used to characterize the core technical parameters and physical properties of the photovoltaic modules, specifically including but not limited to: key parameter information such as module weight, module size, maximum power voltage, maximum power current, and output power.

[0029] For each text region obtained by OCR processing in the nameplate image, a preset field category label is assigned. The field category label is used to indicate the type of key information to which the text content belongs; when the text content does not match any preset key information field, the corresponding text region is marked as a non-key information category to distinguish background explanatory text, manufacturer logos, or other irrelevant information.

[0030] As Figure 2 shown, in the actual nameplates of retired photovoltaic modules, due to differences in design specifications among different manufacturers, or different nameplate layout and marking habits adopted by the same manufacturer in different production batches and different product models, there are often multiple different text expressions for the same key information field in the nameplate; the above differences may stem from factors such as language usage habits, differences in industry terms, or nameplate space layout limitations.

[0031] For example, for the key information field used to represent output power, its corresponding text content may appear in various forms such as "rated power", "maximum output power", "Pmax", etc. Although their specific text descriptions are different, they all point to the same physical parameter at the semantic level and belong to semantically equivalent expressions.

[0032] The maximum power field may appear in Chinese, English, or a mixed Chinese-English form; in Chinese nameplates, there are usually multiple marking methods for maximum power information, including but not limited to "maximum power", "maximum output power", etc.; in English nameplates, common marking methods for maximum power include "Maximum Power", "Power", "Rated Maximum Power", etc., and may also be represented by abbreviated forms such as "Pmax" or "Pm".

[0033] For the weight field, its Chinese expression forms usually include "module weight", "weight", etc., and the English expression form is usually "Weight".

[0034] For the size field, its Chinese expression forms usually include "module size", "size", etc., and the English expression forms include but not limited to "Size", "Dimensions (mm)", and "Dimensions", etc.

[0035] For the maximum power voltage field, its textual expression usually includes different descriptions such as "Voltage at Pmax", "Maximumpower voltage", and "Vmp".

[0036] For the maximum power current field, its textual expression usually includes different descriptions such as "Current at Pmax", "Maximumpower current", and "Imp".

[0037] The tool Label Studio was used to manually annotate the text and its spatial location information obtained after OCR processing, and an annotated dataset was built for training multimodal models. In this embodiment, when annotating the text regions obtained by OCR processing, the judgment is not based solely on the literal form of the text or the specific character content, but rather on a comprehensive analysis that combines the actual semantic meaning expressed by the text. For different text expressions that represent the same physical parameter or the same technical indicator at the semantic level, they are uniformly mapped and labeled with the same preset field category label, thereby achieving unified classification of semantically equivalent texts.

[0038] By using the above-mentioned semantic consistency-based annotation method, the same key information field can obtain consistent category annotation when facing nameplates of multiple languages, manufacturers, and formats. This effectively reduces the problem of field category confusion caused by differences in text expression, language, or abbreviation, thereby improving the semantic modeling ability of the multimodal model in the process of understanding the nameplate layout and extracting key information, and further enhancing the overall accuracy and robustness of the model in recognizing key information on retired photovoltaic module nameplates.

[0039] After completing the annotation, the images of retired photovoltaic nameplates and the annotation information documents are used to build a dataset. The data information is saved in XML and divided into dataset, test set and validation set. Step 2: Based on the data samples, perform semantic modeling on the extracted text characters to obtain text semantic features, model the text spatial location to construct text layout structure features, and at the same time construct visual appearance features and visual layout structure features based on the nameplate image to form a multimodal feature representation. like Figure 6 As shown, this invention proposes a multimodal feature fusion mechanism based on textual semantic features, visual appearance features, and spatial location information, which is used to uniformly model and understand key information in the nameplate images of retired photovoltaic modules.

[0040] The fusion mechanism takes the original image of the nameplate as input, extracts features from the text modality and the image modality respectively, and maps the multimodal features to the same feature space by using a unified position encoding and sequence modeling method to construct a multimodal joint input sequence E, which is then input into a multimodal Transformer network for joint modeling.

[0041] The text content obtained by OCR recognition is segmented into words or characters to obtain several text tokens, denoted as: ; The text token is input into the Word Embedding layer, which uses a pre-trained language model based on the Transformer architecture for text semantic encoding, such as the RoBERTa text encoding model. The Word Embedding layer models the contextual semantics of the input text token through a multi-layer Transformer encoding network, mapping the token to a contextual semantic vector and generating the corresponding text semantic embedding vector. ; Compared with traditional static word vector representation, the text embedding structure based on RoBERTa can combine contextual semantic information to dynamically model the semantic relationship between field names, field units and field values, so that semantically related or context-related text tokens have higher semantic relevance in the feature space.

[0042] Contextual semantic information refers to the semantic relationship between the current text token and its adjacent text tokens. For example, in parameter fields such as “Open Circuit Voltage”, “Voc”, and “38.8V”, the text embedding layer can learn the relationship between field names, field abbreviations, and field values ​​by combining contextual semantic relationships, thereby enhancing the model’s ability to understand semantics and providing basic semantic features for the model to distinguish different types of key information.

[0043] Simultaneously, for each text token, based on its position in the text region of the original nameplate image, a 2D embedding vector is generated using 2D position embedding. Used to characterize the spatial distribution of the text on the nameplate; In the specific implementation, the bounding box information of the text region in the original nameplate image is first obtained. The bounding box is determined by the coordinates of the center point of the text region. and its width and height express; To eliminate the influence of different nameplate image resolutions on positional features, the coordinate information is adjusted according to the width of the nameplate image.W With height H Normalization is performed to obtain normalized location features: ; Subsequently, the normalized location features are input into the location embedding mapping layer, which maps them into a two-dimensional location embedding vector consistent with the text semantic embedding dimension. It is used to characterize the spatial position of the text token in the nameplate and its relative layout relationship, so as to comprehensively describe the spatial distribution of the text token in the nameplate image.

[0044] To characterize the sequential relationship of text tokens in the input sequence, one-dimensional sequence position embedding is also used for each text token to generate a one-dimensional sequence position embedding vector. , is used to indicate its relative order in a multimodal joint sequence.

[0045] By using position embedding vectors, the model can perceive the relative spatial relationship between text tokens when learning features, thereby distinguishing field information that is similar or identical at the semantic level in different areas of the nameplate, and improving the model's ability to understand the overall layout structure of the nameplate.

[0046] like Figure 6 As shown in the lower left corner, the original image of the nameplate is divided into several fixed-size image patches according to preset rules. Each image patch corresponds to a local region in the image, denoted as: ; Each image patch obtained from the segmentation is vectorized and expanded, then input into the linear embedding mapping layer. This maps the corresponding image patch into a fixed-dimensional visual patch embedding vector, denoted as . ; The purpose of this step is to break down a whole 2D image into multiple local regions, so that the image can be organized into a sequence and input into the Transformer like a text token.

[0047] The linear embedding mapping layer is used to map pixel information in image patches to a unified feature space to characterize the local visual appearance features of the corresponding image region. Each image patch is originally a three-dimensional pixel matrix, for example: ; Here, 16×16 represents the height and width of the image block; 3 represents the three color channels of RGB.

[0048] The vectorization unpacking process involves flattening this three-dimensional matrix into a one-dimensional vector in sequence: ; That is, 16 × 16 × 3 = 768.

[0049] The purpose of this step is to convert two-dimensional image patches into one-dimensional numerical vectors that can be processed by the linear embedding mapping layer in a neural network.

[0050] The linear embedding mapping layer is essentially a trainable fully connected layer used to map one-dimensional pixel vectors of image patches into fixed-dimensional visual feature vectors. ; in, Indicates the first m Visual block embedding vectors of image patches; This represents the linear mapping matrix, which is a learnable parameter during model training; This represents the bias term, which is also a learnable parameter; Indicates the first m The unfolded one-dimensional pixel vector of an image patch; the purpose of this step is to convert the original pixel information into a fixed-dimensional visual token, so that the image patch can be input into the multi-modal Transformer along with the text token.

[0051] After the above processing, all image patches are processed to obtain a visual patch embedding vector sequence: ; The visual block embedding vector here can be understood as the vector representation of a local region in the feature space of an image.

[0052] It is similar to a token in text, except that a text token represents a character or word, while a visual block embedding vector represents an image block.

[0053] Visual feature vectors are used to characterize the visual information related to the text in the nameplate image, including but not limited to: the font shape of the text area, the character layout features, the character thickness, the background texture, and the overall image appearance features, thereby supplementing the visual differences that cannot be fully expressed by relying solely on text semantic information and spatial location information.

[0054] By directly modeling the overall visual features of the original nameplate image, the problem of losing contextual information that may be caused by cropping and processing character images one by one in traditional methods is avoided. This effectively preserves the overall visual relationship between text and background, and between text and layout, providing more complete and continuous input information for joint feature learning of multimodal models.

[0055] For each image patch, a 2D embedding vector is generated using the corresponding 2D positional encoding based on its spatial position in the original nameplate image. This is used to characterize the relative position of the image block within the overall layout structure.

[0056] In the specific implementation, the bounding box information of the image region in the original nameplate image is first obtained. The bounding box is determined by the coordinates of the center point of the image region. and its width and height express.

[0057] To eliminate the influence of different nameplate image resolutions on positional features, the coordinate information is normalized according to the width W and height H of the nameplate image to obtain normalized positional features:

[0058] Subsequently, the normalized location features are input into the location embedding mapping layer, which maps them into a two-dimensional location embedding vector consistent with the text semantic embedding dimension. It is used to characterize the spatial position of image blocks in the nameplate and their relative layout relationship, so as to comprehensively describe the spatial distribution of image blocks in the nameplate image.

[0059] Similarly, a one-dimensional sequence position code is assigned to each image patch in the multimodal joint sequence to generate a one-dimensional sequence position embedding vector. Used to identify the order of image patches in the joint input sequence.

[0060] Step 3: Fuse the text semantic features and text layout structure features to obtain text-side fused features; fuse the visual appearance features and visual layout structure features to obtain visual-side fused features. like Figure 6 As shown, text modal features and image modal features are aligned and fused according to a unified embedding structure.

[0061] For each text token, its corresponding semantic / visual embedding vector, one-dimensional sequence position embedding vector, and two-dimensional position embedding vector are combined to obtain a unified-dimensional multimodal feature representation.

[0062] For the i Each text token has a text-side fusion feature represented as follows: ; in, Used to represent the semantic content of the character itself; Used to represent the sequential relationship of characters in a text sequence; Used to represent the two-dimensional spatial position of characters in a nameplate image.

[0063] This step integrates the semantic information, one-dimensional sequence position information, and two-dimensional spatial position information of the text token, establishing a connection between the character content and its arrangement and spatial layout in the nameplate.

[0064] Through the above fusion, each text token carries corresponding positional distribution information while transmitting character semantic information, thereby achieving joint encoding of character content and page structure information.

[0065] For the j The visual fusion features of each image patch are represented as follows: ; in, Used to characterize local visual features of image patches; Used to characterize the order in which image patches are arranged in an image sequence; Used to represent the two-dimensional spatial coordinate information of image patches in an image.

[0066] This step enables the model to establish a relationship between text regions and image regions within a unified feature space by inputting text modal features and visual modal features into the same feature sequence. This invention encodes the local visual features of image blocks along with their order and spatial location information within the image, enabling image region features to carry corresponding spatial distribution information while transmitting texture information.

[0067] Step 4: Jointly model the text-side fusion features and the visual-side fusion features, establish the correspondence between the text-side fusion features and the visual-side fusion features, and generate a joint feature representation for nameplate information recognition.

[0068] Text-side fusion features and image-side fusion features are concatenated in a preset order to form a multimodal joint input sequence: ; This step is used to organize text modal information and visual modal information into the same input sequence, so that text tokens and image blocks can be associated and transmitted within a unified feature space.

[0069] Among them, text-side features are used to convey character semantics and character position information; visual-side features are used to convey image region texture and spatial distribution information.

[0070] For example, on the nameplate of retired photovoltaic modules, "Open Circuit Voltage", "Voc" and "38.8V" are usually located in the same parameter field area.

[0071] Text-side features can not only characterize the semantic relationship between fields such as "Open Circuit Voltage", "Voc", and "38.8V", but also combine two-dimensional position encoding to describe the spatial layout relationship between field names and field values ​​on the nameplate, so that the model can learn that the numerical content corresponding to the parameter field is "38.8V". Visual-side features are not only used to characterize local character texture information, but also to model the overall layout block structure of the nameplate. In photovoltaic module nameplates, multiple sets of parameter fields are usually distributed in a continuous manner on the left side of the nameplate, forming a field block structure with a unified font style, arrangement order and visual boundary.

[0072] Although OCR recognition results can provide the location coordinates of each text token, it is difficult to fully express the regional affiliation and continuous arrangement features of different fields in the overall visual layout.

[0073] Therefore, by modeling the visual features of the original image region, the model can further learn the visual block consistency, layout rules, and regional organization structure between parameter fields, thereby enhancing the model's ability to understand complex nameplate layout structures. Through joint modeling of text-side features and visual-side features, the model can not only understand the semantic content of field names and field values, but also establish field associations by combining the spatial layout relationship of the corresponding region with visual appearance features, thereby achieving accurate identification of key information fields.

[0074] By using serialization and concatenation, the model can simultaneously receive textual and visual information during subsequent encoding processes, thereby establishing a correspondence between character regions and image regions.

[0075] The multimodal joint input sequence is denoted as , which serves as the input to the multimodal Transformer network.

[0076] By jointly modeling text-side fusion features and visual-side fusion features through a multimodal modeling mechanism, a multimodal joint input sequence is formed. The correspondence between text semantic information, spatial layout information and visual appearance information is established, and a joint feature representation for nameplate information recognition is generated to achieve complementary enhancement of information from different modalities, thereby improving the ability to recognize and understand key information of nameplates with complex layouts, multiple languages ​​and multiple expressions.

[0077] The multimodal joint modeling network adopts a Transformer-based encoding structure, including multiple stacked multi-head self-attention layers and feed-forward neural network (FFN) layers, for joint feature learning of textual and visual modal features.

[0078] In the specific implementation process, the multimodal joint input sequence The data is fed into a multi-layer Transformer network, where a multi-head self-attention mechanism is used to jointly model the relationships between different text tokens, between different image patches, and between text tokens and image patches.

[0079] Multi-head self-attention mechanisms calculate attention weights between different features using query vectors, key vectors, and value vectors. The calculation process is as follows:

[0080] in, It is a query vector; It is a key vector; It is a value vector; It is the dimension of the key vector; It is a normalization function used to generate attention weights.

[0081] Through attention computing, the model can learn the relationships between different text tokens, different image patches, and text tokens and image patches, thereby establishing cross-modal correspondences between field names, field abbreviations, and field values.

[0082] For example, for parameter fields such as “Open Circuit Voltage”, “Voc”, and “38.8V”, the model can combine textual semantic information, spatial layout relationships, and visual region features to learn the relationship between the field name region and the corresponding field value region.

[0083] In a multi-head self-attention structure, different attention heads can learn the interaction between text semantic features, layout structure features, and visual appearance features from different feature subspaces, thereby enhancing the model's ability to understand complex nameplate layout structures.

[0084] During the self-attention calculation process, the model can adaptively adjust the information interaction intensity between text semantic features, spatial layout features and visual appearance features according to the degree of correlation between different field regions, thereby enhancing the correlation modeling ability between field name regions and field value regions.

[0085] Feedforward neural network layers are used to perform nonlinear feature mapping and feature enhancement on joint features. Through the layer-by-layer stacking of multi-layer multimodal joint modeling networks, the model can learn local character texture features and spatial adjacency relationships in shallow networks, and further learn the overall layout organization relationship and cross-regional semantic association relationship between parameter fields in deep networks.

[0086] During training, multimodal data samples containing text content, text spatial location information, and corresponding visual features of image regions are used as network inputs, and manually labeled field category labels are used as supervision signals. Supervised learning guides the model to learn the relationship between multimodal features and field categories.

[0087] During the training phase, the multimodal joint modeling network outputs the field category prediction results corresponding to each text token. The prediction results are used to indicate whether the text token belongs to the key information field type or the non-key information category.

[0088] Based on the difference between the model output and the corresponding real field category labels, a loss function is constructed, and the model parameters are iteratively updated through the backpropagation algorithm, so that the model can gradually optimize the joint representation ability between text semantic features, page structure features and visual features.

[0089] Through multiple rounds of iterative training, the model can automatically learn the distribution patterns of key information fields in different layouts, languages, and manufacturers' nameplates, thereby achieving accurate identification of the field categories of different text regions in the nameplate image during the inference stage.

[0090] Through the above training process, the multimodal joint modeling network is able to accurately model and identify key information fields in the nameplates of retired photovoltaic modules by simultaneously utilizing textual semantic information, spatial layout information, and visual appearance information in complex nameplate scenarios.

[0091] Compared to methods based solely on single text features or traditional OCR post-processing rules, this embodiment significantly improves the model's adaptability and recognition accuracy to nameplates with diverse languages, layouts, and semantic expressions through multimodal joint modeling training.

[0092] Step 5: Identification and output of key information fields.

[0093] After completing the multimodal joint modeling, the output layer set at the end of the multimodal joint modeling network performs inference calculations on the input multimodal joint input sequence and outputs the field category prediction results corresponding to each text token or text region. The prediction results are used to indicate the key information field type or non-key information category to which the text token or text region belongs. Figure 7 This is a schematic diagram illustrating the process of recognizing the nameplate image of a retired photovoltaic module and visualizing the recognition result using borders and category labels in an embodiment of the present invention.

[0094] Based on the field category prediction results, text tokens or text regions belonging to the same field category in the nameplate image are classified and aggregated. Field names, field units, and field values ​​that are scattered in different locations are unified and integrated to form a complete field semantic description.

[0095] To verify the effectiveness of the multimodal joint modeling method proposed in this embodiment, a comparative experiment was conducted between the method of this embodiment and the character recognition method based on local field region clipping under the same retired photovoltaic module nameplate dataset and the same evaluation index.

[0096] Traditional methods typically begin by locating the field name regions in the nameplate. For example, after detecting field names such as "Voc", the entire parameter row containing the field name is extracted. Then, an OCR model is used to perform character recognition on the extracted local field image to obtain the corresponding parameter values.

[0097] In contrast, this invention directly constructs a joint input sequence of text and visual modalities based on the original nameplate image, and uses a multimodal Transformer network to perform overall joint modeling of text semantic information, spatial layout information and visual appearance information, thereby establishing the association between field names, field units and field values.

[0098] Experimental results show that the traditional local field region cropping recognition method has a recognition accuracy of 73.08% on the validation set, while the field recognition accuracy of the model in this embodiment reaches 98.17%, the F1 value reaches 86.57%, and the precision and recall are both maintained at 86.57%, with the validation loss value converging to 0.074.

[0099] The above results demonstrate that this embodiment can effectively improve recognition accuracy through multimodal joint modeling.

[0100] Specifically, for the same key information field, multiple text tokens corresponding to it are concatenated or combined according to their spatial or logical order in the nameplate to obtain a structured field information result. The structured field information includes at least the field name and its corresponding field value content, such as component weight, component size, maximum power voltage, maximum power current and output power, etc.

[0101] The identified structured field information is associated with the corresponding nameplate image identification information and written into the database according to the preset data structure to achieve persistent storage and unified management of key information fields.

[0102] The database can be a relational database or a non-relational database. The field table structure should include at least the nameplate image number, field type identifier, field value, and field unit, to support subsequent data query, statistical analysis, and system calls.

[0103] By storing structured field information in a database, the recognition results can directly serve business applications in the decommissioned photovoltaic module management system, such as module information filing, performance evaluation, life analysis, and recycling decisions, thereby improving the overall system's automation level and engineering practicality.

[0104] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for decommissioned photovoltaic module label identification, the method comprising: receiving a decommissioned photovoltaic module; and identifying a label on the decommissioned photovoltaic module. Includes the following steps: Step 1: Collect images of retired photovoltaic modules and extract text characters and spatial location information from the nameplate area; Step 2: Perform semantic modeling on the text characters to obtain text semantic features; The spatial location of the text is modeled to obtain the text layout structure features; visual appearance features and visual layout structure features are constructed using the nameplate area image; multimodal features are input into the recognition model to output key information.

2. The method for decommissioned photovoltaic module label identification of claim 1, wherein, Step two specifically includes: Step 21: Divide the text content into several tokens by word segmentation or character-level segmentation, and divide the nameplate image into several image blocks; Step 22: Process the token using a text embedding layer to generate a text semantic embedding vector. ; Step 23: Process the text region positions of the token and image block using two-dimensional positional encoding to generate 2D text positional embedding vectors respectively. and 2D image patch location embedding vector ; Step 24: Process the text region positions of the token and image patch using one-dimensional sequence position encoding to generate a 1D text sequence embedding vector. and 1D image patch sequential embedding vector ; Step 25: Vectorize and expand the image patch, input it into the linear embedding mapping layer, and generate visual patch embedding vectors. ; Step 26, place the first i Text token fusion ; Step 27, place the first j Image patch fusion ; Step 28, and By concatenating the features, multimodal features are obtained. .

3. The method for identifying tags on decommissioned photovoltaic modules according to claim 2, characterized in that, The formula is: ； in, Represents a linear mapping matrix; Indicates the bias term. For the first j Image blocks, : No. j The one-dimensional pixel vector formed by unfolding an image patch.

4. The method for identifying tags on decommissioned photovoltaic modules according to claim 1, characterized in that, The recognition model includes: Transformer network.

5. The method for identifying tags on decommissioned photovoltaic modules according to claim 1, characterized in that, Image preprocessing includes cropping, orthogonalization, and angle correction.

6. The method for identifying tags on decommissioned photovoltaic modules according to claim 1, characterized in that, Spatial location information includes: boundary coordinates, area width, and area height.

7. The method for identifying tags on decommissioned photovoltaic modules according to claim 1, characterized in that, Text characters include: Component weight, component size, maximum power voltage, maximum power current, and output power.

8. A label identification system for decommissioned photovoltaic modules, characterized in that, include: Memory is used to store instructions that can be executed by the processor; A processor for executing instructions to implement the method for identifying tags of decommissioned photovoltaic modules as described in any one of claims 1-7.

9. A computer-readable medium storing computer program code, characterized in that, The computer program code, when executed by a processor, implements the tag identification method for decommissioned photovoltaic modules as described in any one of claims 1-7.

Images