Photovoltaic information analysis method and device and storage medium

Abstract

The invention discloses a photovoltaic information analysis method and device and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring original data related to the photovoltaic industry from a data source; extracting first text information from the text data of the original data; and, when the original data includes the image data, extracting second text information from the image data; identifying the category of the original data according to the first text information and the second text information; and under the condition that the category of the original data is the target category, generating summary information of the original data. According to the embodiment of the invention, the key information is extracted from the multi-source and multi-mode data, various data forms in the photovoltaic industry can be covered, the problem of information omission caused by single data type is reduced, the type of the original data can be accurately identified, the summary information is generated after the original data is confirmed to belong to the target type, and the data processing efficiency is improved. The key content is further extracted, and the analysis efficiency and accuracy of the photovoltaic information are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a photovoltaic information analysis method, device, and storage medium. Background Art

[0002] In recent years, the photovoltaic industry, a core force in the clean energy sector, has experienced rapid growth and is driving the transformation of the global energy mix. With increased policy support from various countries, the cost of photovoltaic power generation has significantly decreased, and the market size continues to expand. At the same time, photovoltaic technology has gradually expanded from single component production to the entire industry chain, encompassing power generation system optimization, energy storage technology integration, and intelligent operations management, forming a complex and dynamic industry ecosystem.

[0003] In this process, the industry's demand for information acquisition and decision support is becoming increasingly urgent. Efficient analysis of unstructured data, such as news, research reports, and policy documents, is particularly crucial given the coexistence of multilingual and multi-source data. However, most companies currently rely on manual screening or rule-based analysis tools to analyze this type of data, making it difficult to quickly and accurately extract key information. Summary of the Invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a photovoltaic information analysis method, device and storage medium to improve the efficiency and accuracy of photovoltaic information analysis.

[0005] In a first aspect, the present application provides a photovoltaic information analysis method, comprising:

[0006] Obtain raw data related to the photovoltaic industry from data sources;

[0007] extracting first text information from the text data of the original data; and, in a case where the original data includes image data, extracting second text information from the image data;

[0008] identifying a category of the original data according to the first text information and the second text information;

[0009] When the category of the original data is the target category, summary information of the original data is generated.

[0010] The photovoltaic information analysis method provided in the embodiment of the present application obtains raw data related to the photovoltaic industry from a data source; extracts first text information from the text data of the raw data; and, in the case where the raw data includes image data, extracts second text information from the image data; identifies the category of the raw data based on the first text information and the second text information; and generates summary information of the raw data when the category of the raw data is a target category. The embodiment of the present application can cover various data forms of the photovoltaic industry by extracting key information from multi-source and multi-modal data, reducing the problem of information omission caused by a single data type, thereby being able to accurately identify the category of the raw data, generate summary information after confirming that the raw data belongs to the target category, further refine key content, and provide users with accurate and efficient information output, facilitating a quick grasp of the core dynamics of the photovoltaic industry, and improving the efficiency and accuracy of photovoltaic information analysis.

[0011] According to one embodiment of the present application, before extracting the first text information from the text data of the original data, the method includes:

[0012] performing language recognition on the raw data;

[0013] In a case where the original data includes data in a language other than the target language, the original data is translated into the target language.

[0014] In this embodiment, by performing language recognition on the original data, the language type of the data can be quickly determined. When non-target language data is identified, it is converted into the target language, thereby improving the integrity of the information and reducing information omissions due to language barriers. This multilingual processing capability enables photovoltaic information analysis to cover a wider range of data sources. Especially in the context of globalization, multilingual data in the photovoltaic industry is increasing. This solution can effectively integrate text information in different languages, further improving the efficiency and accuracy of photovoltaic information analysis.

[0015] According to one embodiment of the present application, before extracting the first text information from the text data of the original data, the method includes:

[0016] Performing deduplication processing on the original data to delete duplicate data in the original data;

[0017] The original data after deduplication processing is compared with the historical data in a preset database, and the data in the original data after deduplication processing that is duplicated with the historical data is deleted.

[0018] In this embodiment, by deduplicating the original data and deleting the duplicate data therein, redundant information can be effectively reduced, and resource waste caused by duplicate data in subsequent analysis can be reduced. The deduplicated original data is compared with the historical data in the preset database, and the parts that are duplicated with the historical data are further deleted, thereby improving the novelty and timeliness of the information and reducing repeated analysis of the processed data.

[0019] According to one embodiment of the present application, extracting the first text information from the text data of the original data includes:

[0020] The text data is input into a large language model, and data associated with target content in the text data is extracted using the large language model to obtain first text information.

[0021] In this embodiment, by inputting text data into a large language model and utilizing the model's powerful language understanding and generation capabilities, data associated with the target content is extracted to obtain the first text information. This reduces the problems of omissions or errors caused by imperfect rules in traditional methods and improves the accuracy and efficiency of information extraction.

[0022] According to one embodiment of the present application, extracting the second text information from the image data includes:

[0023] In the case where the image data includes text content, converting the text content into second text information through optical character recognition;

[0024] In a case where the image data includes chart information, the chart information is converted into second text information by using a large language model.

[0025] In this embodiment, the text content in the image data is converted into an editable text format through optical character recognition technology, which can improve the integrity of the text data; for the chart information in the image, the data, trends and key information in the chart can be converted into clear text expressions through a large language model. This targeted processing of different types of information in the image enriches the data source and further enhances the accuracy of photovoltaic information analysis.

[0026] According to one embodiment of the present application, the method further includes:

[0027] If the original data includes a file attachment, parsing the file attachment to obtain text content of the file attachment;

[0028] The text content of the file attachment is added to the first text message.

[0029] In this embodiment, by parsing the file attachment and adding its text content to the first text information, the data source for photovoltaic information analysis is further enriched, and the integrity and accuracy of the information are improved.

[0030] According to one embodiment of the present application, identifying the category of the original data based on the first text information and the second text information includes:

[0031] Fine-tune the large language model to obtain a classification model;

[0032] The first text information and the second text information are input into a classification model to obtain the category of the original data output by the classification model.

[0033] In this embodiment, by fine-tuning the large language model to construct a classification model, and using the model to classify the first text information and the second text information, the powerful semantic understanding and few-shot learning capabilities of the large language model are utilized, and rapid modeling can be achieved with a small amount of sample data, reducing the problem of traditional methods' dependence on massive labeled data. This enables the model to quickly respond to the specific context and classification requirements of the photovoltaic industry, further improving the efficiency and accuracy of photovoltaic information analysis.

[0034] In a second aspect, the present application provides a photovoltaic information analysis device, the device comprising:

[0035] The acquisition module is used to obtain raw data related to the photovoltaic industry from data sources;

[0036] an extraction module, configured to extract first text information from the text data of the original data; and, if the original data includes image data, extract second text information from the image data;

[0037] an identification module, configured to identify a category of the original data according to the first text information and the second text information;

[0038] A generating module is used to generate summary information of the original data when the category of the original data is the target category.

[0039] The photovoltaic information analysis device provided in the embodiment of the present application obtains raw data related to the photovoltaic industry from a data source; extracts first text information from the text data of the raw data; and, in the case where the raw data includes image data, extracts second text information from the image data; identifies the category of the raw data based on the first text information and the second text information; and generates summary information of the raw data when the category of the raw data is a target category. The embodiment of the present application can cover various data forms of the photovoltaic industry by extracting key information from multi-source and multi-modal data, reducing the problem of information omission caused by a single data type, thereby being able to accurately identify the category of the raw data, generate summary information after confirming that the raw data belongs to the target category, further refine key content, and provide users with accurate and efficient information output, making it easier to quickly grasp the core dynamics of the photovoltaic industry and improving the efficiency and accuracy of photovoltaic information analysis.

[0040] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the photovoltaic information analysis method as described in the first aspect above is implemented.

[0041] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the photovoltaic information analysis method as described in the first aspect above.

[0042] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the photovoltaic information analysis method as described in the first aspect above.

[0043] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0044] The photovoltaic information analysis method provided in the embodiment of the present application obtains raw data related to the photovoltaic industry from a data source; extracts first text information from the text data of the raw data; and, in the case where the raw data includes image data, extracts second text information from the image data; identifies the category of the raw data based on the first text information and the second text information; and generates summary information of the raw data when the category of the raw data is a target category. The embodiment of the present application can cover various data forms of the photovoltaic industry by extracting key information from multi-source and multi-modal data, reducing the problem of information omission caused by a single data type, thereby being able to accurately identify the category of the raw data, generate summary information after confirming that the raw data belongs to the target category, further refine key content, and provide users with accurate and efficient information output, facilitating a quick grasp of the core dynamics of the photovoltaic industry, and improving the efficiency and accuracy of photovoltaic information analysis.

[0045] Furthermore, by performing language recognition on the raw data, the language type of the data can be quickly determined. When non-target language data is identified, it is converted into the target language, improving the integrity of the information and reducing information omissions due to language barriers. This multilingual processing capability enables photovoltaic information analysis to cover a wider range of data sources. Especially in the context of globalization, multilingual data in the photovoltaic industry is increasing. This solution can effectively integrate text information in different languages, further improving the efficiency and accuracy of photovoltaic information analysis.

[0046] Furthermore, by deduplicating the original data and deleting the duplicate data, it is possible to effectively reduce redundant information and reduce the waste of resources caused by duplicate data in subsequent analysis. The deduplicated original data is compared with the historical data in the preset database, and the parts that are duplicated with the historical data are further deleted, which improves the novelty and timeliness of the information and reduces the repeated analysis of the processed data.

[0047] Furthermore, by inputting text data into a large language model and utilizing the model's powerful language understanding and generation capabilities, data associated with the target content is extracted to obtain the first text information, reducing the problems of omissions or errors caused by imperfect rules in traditional methods and improving the accuracy and efficiency of information extraction.

[0048] Furthermore, optical character recognition technology can be used to convert text content in image data into an editable text format, which can improve the integrity of text data. For chart information in images, large language models can be used to convert the data, trends and key information in the charts into clear text expressions. This targeted processing of different types of information in images enriches the data source and further enhances the accuracy of photovoltaic information analysis.

[0049] Furthermore, by parsing the file attachment and adding its text content to the first text information, the data source for photovoltaic information analysis is further enriched, and the integrity and accuracy of the information are improved.

[0050] Furthermore, by fine-tuning the large language model to build a classification model, and using this model to classify the first text information and the second text information, the powerful semantic understanding and few-shot learning capabilities of the large language model are utilized, and rapid modeling can be achieved with a small amount of sample data, reducing the traditional method's dependence on massive labeled data. This enables the model to quickly respond to the specific context and classification needs of the photovoltaic industry, further improving the efficiency and accuracy of photovoltaic information analysis.

[0051] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0053] Figure 1 Schematic diagram of the photovoltaic information analysis method provided in the embodiment of the present application;

[0054] Figure 2 is a schematic diagram of a scenario example provided in an embodiment of the present application;

[0055] Figure 3 Schematic diagram of the structure of the photovoltaic information analysis device provided in an embodiment of the present application;

[0056] Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0058] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0059] The photovoltaic industry is a key component of the global energy transition. With the continuous advancement of solar technology and increased policy support, the PV industry is rapidly developing worldwide. PV technology has not only significantly increased power generation, but has also achieved promising results in energy storage, distributed generation, and other fields. This rapid development has made obtaining, analyzing, and interpreting news and information about the industry increasingly complex.

[0060] In the field of photovoltaic industry information analysis, existing technical solutions primarily focus on manual screening or rule-based analysis tools. However, PV industry news and reports contain a vast amount of information, including domain terminology, policies and regulations, and market trends. This data often suffers from irregular formats and inaccurate semantics. Faced with this massive and constantly updated volume of PV industry news and reports, it is difficult for humans to quickly filter out valuable information. Furthermore, manual screening is prone to subjective bias, as different people have different criteria for the importance and relevance of information, which can lead to omissions or misjudgments of key information. Verification and collation require significant time, increasing workload and the potential for error. Rule-based analysis tools rely on expert experience and historical data to formulate rules. The rapid development of the PV industry has led to a constant emergence of new terminology, policies and regulations, and market trends, making it difficult to keep rules up to date and cover all situations. Furthermore, rule-based analysis tools struggle to accurately parse the underlying semantics of PV news, especially in multilingual or complex contexts. Therefore, manual screening or rule-based analysis tools are often time-consuming and lack accuracy.

[0061] To solve at least one of the above technical problems, the present invention provides a photovoltaic information analysis method, device, and storage medium. The photovoltaic information analysis method, device, and storage medium provided by the present invention are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0062] The photovoltaic information analysis method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0063] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0064] In the following embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse, and a joystick.

[0065] The photovoltaic information analysis method provided in the embodiment of the present application can be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablets, computers, cameras and wearable devices, etc. The photovoltaic information analysis method provided in the embodiment of the present application is described below using an electronic device as an example of the execution subject.

[0066] like Figure 1 As shown, the photovoltaic information analysis method includes: step 110, step 120, step 130 and step 140.

[0067] Step 110: Obtain raw data related to the photovoltaic industry from a data source.

[0068] In the embodiments of this application, the data source can be various websites on the internet, such as photovoltaic industry news websites, research institute websites, market analysis report platforms, and websites of photovoltaic-related companies. The raw data can include news reports, research reports, technology releases, market analyses, financial reports, and other data sources. This raw data can provide support for industry research, market forecasts, and policy analysis.

[0069] In some embodiments, raw data related to the photovoltaic industry can be crawled from the internet using web crawler technology. For example, the Scrapy open source crawler framework can be used to select relevant data sources for crawling based on the characteristics of the photovoltaic industry. Scrapy is a fast and advanced web crawler framework that can efficiently crawl website data and extract structured information.

[0070] In one example, a crawler script can be written using the Scrapy framework, defining the crawling start URL (Uniform Resource Locator), parsing rules, and data extraction logic. For example, news headlines, content, and release dates can be extracted from web pages using XPath (XML Path Language) or CSS (Cascading Style Sheets) selectors.

[0071] Step 120 : extracting first text information from the text data of the original data; and, if the original data includes image data, extracting second text information from the image data.

[0072] In the embodiments of the present application, the raw data may include text data, which may come from a variety of sources, such as web page content captured by web crawlers, industry reports, news articles, etc. This text data is usually stored in formats such as HTML (HyperText Markup Language), PDF (Portable Document Format), and TXT, and contains a large amount of valuable information related to the photovoltaic industry.

[0073] In some embodiments, the first text information can be extracted from the text data of the original data by means of keyword extraction, key sentence extraction and entity recognition. Keyword extraction is to find out the words that can represent the theme of the text by analyzing the frequency and importance of words in the text. For example, the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm can be used to calculate the importance of each word, thereby extracting keywords. Key sentence extraction is to find out those sentences that can summarize the main content of the text from the text. Key sentence extraction can be performed by natural language processing technology, such as using a model based on BERT (Bidirectional Encoder Representations from Transformers) or GPT (Generative Pre-trained Transformer) to evaluate the importance of sentences, thereby achieving key sentence extraction. Of course, the first text information can also be extracted from the text data of the original data by other means, and the embodiments of the present application are not limited to this.

[0074] In some embodiments, before extracting the first text information, the text data can also be preprocessed to remove noise and irrelevant content in the text, making it cleaner and easier to process. For example, the preprocessing can include removing HTML tags, script code, and other non-text content in the text, and can also include stop word removal, removing words that frequently appear in the text but contribute less to the semantics, such as "的", "是", "and", "the", etc.

[0075] In the embodiments of the present application, the original data may also include image data, and the image data may contain text information related to the photovoltaic industry, such as charts, posters, document scans, etc. In order to make full use of the information in these image data, it is necessary to extract the text content from the images.

[0076] In some embodiments, the text in the image data can be converted into an editable text format through OCR (Optical Character Recognition) technology to obtain the second text information. The text information in the image data can also be recognized by classifying and object-detecting the image data through deep learning algorithms to obtain the second text information. Of course, the second text information can also be extracted from the image data by other means, and the embodiments of the present application do not limit this.

[0077] In some embodiments, since the text in the image data may be affected by various factors, such as lighting conditions, resolution, noise, etc., these factors may reduce the accuracy of OCR. Before performing OCR, the image data can be preprocessed to improve the quality of the image and make it more suitable for text recognition. For example, the preprocessing can include grayscale processing of the image data, reducing the color information of the image data, highlighting the contrast between the text and the background, thereby improving the accuracy of OCR. The preprocessing can also include binarization processing of the image data to further enhance the contrast between the text and the background. The preprocessing can also include denoising processing of the image data to remove noise points in the image.

[0078] In some embodiments, due to limitations in image quality, text font, or the OCR tool itself, the text content extracted by OCR may contain some errors. Therefore, after obtaining the second text information, the second text can also be post-processed. For example, a language model can be used to check the grammar and spelling errors of the text and perform automatic correction.

[0079] Step 130: Identify the category of the original data according to the first text information and the second text information.

[0080] In the embodiments of this application, by identifying the categories of raw data, it is possible to better organize, manage, and analyze this raw data. For example, based on the nature of photovoltaic industry data, the raw data can be divided into policy and regulations, market analysis, technology research, strategic analysis, macroeconomic situation, industry dynamics, industry chain dynamics, international energy observation, etc. This classification allows for more efficient retrieval and utilization of data, providing support for subsequent decision-making.

[0081] In the embodiments of the present application, the first text information is typically extracted from plain text data, such as news articles, research reports, or web page content. The first text information may include keywords, key sentences, named entities (such as company names, dates, locations, etc.), and the overall theme of the text. The second text information is extracted from image data, such as text in images such as charts, posters, and document scans.

[0082] In some embodiments, before identifying the category of the original data, the first text information and the second text information can be preprocessed, such as text cleaning, stop word removal, text formatting, etc., to obtain high-quality text information to improve the accuracy of category identification of the original data.

[0083] In some embodiments, the first text information and the second text information can be combined into a complete text description. For example, the extracted keywords and key sentences can be combined to form a comprehensive text feature set. The combined text can be classified using a text classification model, such as a support vector machine, a naive Bayes classifier, a gradient boosted decision tree, a random forest, or a deep learning model.

[0084] In some embodiments, the first and second text messages can be categorized separately, and the classification results can then be combined using a voting mechanism or weighted average. For example, if the first text message categorizes the data as market analysis, while the second text message categorizes it as technical research, we can set weights to determine the final classification result. The weights can be adjusted based on factors such as the source, quality, and importance of the data.

[0085] Step 140: When the category of the original data is the target category, generate summary information of the original data.

[0086] In this embodiment of the present application, target categories refer to categories related to the purpose of analysis and can be pre-defined. For example, in photovoltaic industry data, target categories could be "Policies and Regulations," "Market Analysis," "Technical Research," "Strategic Analysis," and so on. When raw data is identified as belonging to a target category, a concise summary can be generated to facilitate further analysis and decision-making, saving time and effort.

[0087] In the embodiments of this application, the summary information is a high-level summary and conclusion of the original data content. It extracts key information from the original data and removes redundant and unimportant content, allowing users to quickly grasp the main points and key points of the data. For example, for a long report on photovoltaic industry policies, the summary information can extract key points such as the policy's core provisions, implementation timeline, and impact scope, helping users quickly obtain important information.

[0088] In some embodiments, a summary can be generated by extracting key sentences or paragraphs from the first and second text information. This approach relies on natural language processing techniques, such as keyword extraction and sentence importance scoring. For example, the TF-IDF algorithm can be used to identify keywords in the text information, and then, based on the distribution of these keywords, sentences containing these keywords are selected as the summary content.

[0089] In some embodiments, a summary can also be generated by using a deep learning model, such as a model of a Transformer architecture. The model will regenerate a fluent and concise summary text based on the input text content. For example, large language models (LLM) such as BERT and GPT can be used to generate summary information. These models can understand the semantics and grammatical structure of the language by learning a large amount of text data, thereby generating a natural and fluent summary. Of course, summary information of the original data can also be generated by other means, and the embodiments of the present application are not limited to this.

[0090] The photovoltaic information analysis method provided in the embodiment of the present application obtains raw data related to the photovoltaic industry from a data source; extracts first text information from the text data of the raw data; and, if the raw data includes image data, extracts second text information from the image data; identifies the category of the raw data based on the first text information and the second text information; and generates summary information of the raw data if the category of the raw data is a target category. The embodiment of the present application can cover various data forms of the photovoltaic industry by extracting key information from multi-source and multi-modal data, reducing the problem of information omission caused by a single data type, thereby accurately identifying the category of the raw data, generating summary information after confirming that the raw data belongs to the target category, further refining key content, and providing users with accurate and efficient information output, facilitating a quick grasp of the core dynamics of the photovoltaic industry, and improving the efficiency and accuracy of photovoltaic information analysis.

[0091] In some embodiments, before extracting the first text information from the text data of the original data, the method includes:

[0092] Perform language identification on raw data;

[0093] In the case where the original data includes data in a language other than the target language, the original data is translated into the target language.

[0094] In this embodiment, since the raw data may come from different data sources and may contain content in multiple languages, the raw data can be language identified by using any technology or tool. Many natural language processing (NLP) libraries and frameworks provide language identification functions. For example, an open source language identification library such as langid.py can be used to perform language detection on the raw data, or the langdetect library or langid library in Python can be used to perform language detection on the raw data. These tools can analyze the character patterns, vocabulary structure and grammatical features of the text in the raw data to determine the language type. In actual applications, the raw data can be input into these language identification tools to obtain the language label of the text.

[0095] After completing language identification, you need to process the raw data that is not in the target language. For example, if the analysis target is Chinese, you can use Chinese as the target language. If the raw data contains text in other languages, you can translate this text into Chinese for unified processing and analysis, fully utilizing the information in multilingual data.

[0096] In this embodiment, the translation of the original data can be achieved through a translation model. For example, the translation model can be a large language model or other neural machine translation model based on deep learning, which is not limited in this embodiment of the application.

[0097] In this embodiment, by performing language recognition on the original data, the language type of the data can be quickly determined. When non-target language data is identified, it is converted into the target language, thereby improving the integrity of the information and reducing information omissions due to language barriers. This multilingual processing capability enables photovoltaic information analysis to cover a wider range of data sources. Especially in the context of globalization, multilingual data in the photovoltaic industry is increasing. This solution can effectively integrate text information in different languages, further improving the efficiency and accuracy of photovoltaic information analysis.

[0098] In some embodiments, before extracting the first text information from the text data of the original data, the method includes:

[0099] Perform deduplication processing on the original data to delete duplicate data in the original data;

[0100] The deduplicated original data is compared with the historical data in the preset database, and the data that is duplicated between the deduplicated original data and the historical data is deleted.

[0101] When processing raw data, you may encounter duplicate data. This duplicate data may originate from redundant capture during data collection or overlap between different data sources. To improve the accuracy and efficiency of subsequent analysis, you can perform deduplication processing on the raw data before extracting the first text information to identify and remove duplicate items.

[0102] In this embodiment, a hash algorithm and / or a similarity calculation method can be used to detect and delete duplicate data. The hash algorithm is an efficient means of deduplication that quickly identifies duplicates by converting data into a hash value of a fixed length. For example, a hash algorithm such as MD5 (Message-Digest Algorithm 5) or SHA-256 (Secure Hash Algorithm 256) can be used to generate a unique hash value for each piece of data in the original data. If the hash values of two pieces of data are the same, then the two pieces of data can be considered to be duplicates.

[0103] The advantage of hashing algorithms is their fast computational speed and suitability for processing large datasets. However, hashing algorithms can only detect identical duplicate data and cannot identify records with similar but not identical content. To compensate for the shortcomings of hashing algorithms, similarity calculation methods can be combined to further detect duplicate data. Similarity calculation methods identify duplicates by comparing the similarity of text content. For example, algorithms such as Jaccard similarity or cosine similarity can be used to calculate the similarity between texts in the original data. If the similarity between two texts exceeds a set threshold, such as 90%, the two pieces of data can be considered duplicates. The advantage of similarity calculation methods is that they can identify records with similar but not identical content, thereby improving the accuracy of deduplication. For example, when processing news reports, different media may report on the same event in a similar manner. Although the content is not exactly the same, these duplicate reports can be identified through similarity calculation.

[0104] In this embodiment, after deduplication of the raw data is completed, in order to further improve the uniqueness of the raw data and reduce the problem of repeated analysis of the same information, the deduplication-processed raw data can be compared with historical data in a preset database. The preset database stores previously processed data, and the historical data can be used as a reference to identify and delete duplicate data already in the database.

[0105] When comparing the original data after deduplication processing with the historical data in the preset database, a method similar to that used in deduplication processing can be used, that is, a hash algorithm and / or a similarity calculation method can be used to detect and delete duplicate data.

[0106] In this embodiment, by deduplicating the original data and deleting the duplicate data therein, redundant information can be effectively reduced, and resource waste caused by duplicate data in subsequent analysis can be reduced. The deduplicated original data is compared with the historical data in the preset database, and the parts that are duplicated with the historical data are further deleted, thereby improving the novelty and timeliness of the information and reducing repeated analysis of the processed data.

[0107] In some embodiments, extracting the first text information from the text data of the original data includes:

[0108] The text data is input into the large language model, and the data associated with the target content in the text data is extracted by the large language model to obtain the first text information.

[0109] Large Language Models (LLMs) are models trained using deep learning techniques that can understand, generate, and process human language. LLMs typically have hundreds of millions to trillions of parameters and are capable of performing complex tasks such as semantic understanding, text generation, and question answering.

[0110] In this embodiment, any large language model, such as a GPT series model, a BERT model, etc., can be selected to extract data associated with the target content in the text data to obtain the first text information.

[0111] Specifically, after selecting a large language model, the text data of the original data can be input into the large language model. The large language model uses its internal neural network structure, pre-trained knowledge and language patterns to understand the semantics and contextual relationships of the text, thereby determining which information is relevant to the target content. For example, the analysis goal is to extract information related to technological innovation from research reports on the photovoltaic industry, and the target content is information related to technological innovation in the photovoltaic industry. The large language model will identify keywords related to technological innovation in the photovoltaic industry by analyzing the vocabulary and semantic relationships in the text, such as "heterojunction battery", "perovskite technology", "conversion efficiency improvement", etc., and extract sentences or paragraphs containing these keywords as information related to the target content, thereby obtaining the first text information.

[0112] In this embodiment, by inputting text data into a large language model and utilizing the model's powerful language understanding and generation capabilities, data associated with the target content is extracted to obtain the first text information. This reduces the problems of omissions or errors caused by imperfect rules in traditional methods and improves the accuracy and efficiency of information extraction.

[0113] In some embodiments, extracting the second text information from the image data includes:

[0114] In the case where the image data includes text content, converting the text content into second text information by optical character recognition;

[0115] In the case where the image data includes chart information, the chart information is converted into second text information by the large language model.

[0116] In this embodiment, it is possible to identify whether the image data contains text content. If the image data contains text content, such as text in a document scan, poster, or chart, the characters in the image data can be identified through OCR and converted into corresponding text information.

[0117] In this embodiment, in addition to text content, the image data may also contain chart information, such as bar charts, line charts, pie charts, etc. These charts are usually used to display data and trends. In order to extract valuable information from the charts, the chart content can be converted into text form for further analysis and processing.

[0118] Specifically, the large language model's generative capabilities can be used to describe the content of a chart as natural language text. For example, for a line chart showing the growth of photovoltaic installed capacity, the large language model can generate a description like this: "From 2020 to 2024, global photovoltaic installed capacity showed a significant growth trend. The installed capacity in 2020 was 100GW, and it increased to 300GW by 2024." In this way, the visual information in the chart can be converted into textual information that is easy to understand and analyze.

[0119] In this embodiment, the text content in the image data is converted into an editable text format through optical character recognition technology, which can improve the integrity of the text data; for the chart information in the image, the data, trends and key information in the chart can be converted into clear text expressions through a large language model. This targeted processing of different types of information in the image enriches the data source and further enhances the accuracy of photovoltaic information analysis.

[0120] In some embodiments, the method further comprises:

[0121] In the case where the original data includes a file attachment, the file attachment is parsed to obtain the text content of the file attachment;

[0122] The text content of the file attachment is added to the first text message.

[0123] In this embodiment, the original data may include file attachments, which can exist in various formats, such as PDF, Word document, Excel spreadsheet or other common file types. File attachments usually include additional information relevant to the original data. File attachments in the original data can be identified by checking the original information of the original data. For example, if the original data comes from an email or a web page, file attachments usually have clear file names and extensions, such as .pdf, .docx or .xlsx. By parsing these file names and extensions, file attachments and the formats of the file attachments can be identified.

[0124] After identifying the file attachment, the file attachment can be parsed to obtain the text content of the file attachment. Specifically, a corresponding tool or method can be selected to parse the file attachment based on the format of the file attachment. For example, a PDF file may contain scanned image text, and OCR technology can be used to identify and extract the text content in the PDF file and add the text content to the first text information.

[0125] In this embodiment, by parsing the file attachment and adding its text content to the first text information, the data source for photovoltaic information analysis is further enriched, and the integrity and accuracy of the information are improved.

[0126] In some embodiments, identifying the category of the original data based on the first text information and the second text information includes:

[0127] Fine-tune the large language model to obtain a classification model;

[0128] The first text information and the second text information are input into the classification model to obtain the category of the original data output by the classification model.

[0129] Because large language models are typically trained based on general corpora and may not fully adapt to the semantics and logic of a specific domain, directly using a pre-trained large language model may not achieve optimal results. Therefore, fine-tuning the large language model is necessary to better adapt it to task requirements.

[0130] In this embodiment, a large language model can be fine-tuned in combination with few-shot learning to obtain a classification model. Specifically, a small number of photovoltaic industry-related information samples with accurate and well-labeled categories can be used. These information samples should cover as many different categories as possible to ensure the balance of model learning. A clear and concise prompt can also be designed to embed the labeled samples. For example, a prompt in the following format can be designed:

[0131] Input: This is a news item about policy adjustments in the photovoltaic industry.

[0132] Output: Category: Policy Developments

[0133] Input: A photovoltaic company announced the launch of a new generation of high-efficiency photovoltaic modules.

[0134] Output: Category: Technological Innovation

[0135] The designed prompt is input into the model and fine-tuned. During fine-tuning, the large language model updates its internal parameters based on a small amount of labeled data to better suit the task of classifying photovoltaic industry information. This allows the large language model to quickly understand the characteristics of different types of information. During fine-tuning, the prompt can also be optimized. For example, the format or content of the prompt can be adjusted based on the output of the large language model to better align it with its understanding.

[0136] After fine-tuning is completed, the large language model can be determined as a classification model, and the first text information and the second text information can be input into the classification model to obtain the category of the original data output by the classification model.

[0137] In this embodiment, by fine-tuning the large language model to construct a classification model, and using the model to classify the first text information and the second text information, the powerful semantic understanding and few-shot learning capabilities of the large language model are utilized, and rapid modeling can be achieved with a small amount of sample data, reducing the problem of traditional methods' dependence on massive labeled data. This enables the model to quickly respond to the specific context and classification requirements of the photovoltaic industry, further improving the efficiency and accuracy of photovoltaic information analysis.

[0138] The photovoltaic information analysis method provided by the embodiments of the present application is described below using a scenario example. In this scenario example, a crawler program can first be used to crawl raw data from different data sources. After collecting the raw data, the language of the raw data can be identified. If the raw data includes non-Chinese languages, the raw data can be translated into Chinese using a translation model. If the raw data is entirely in Chinese, a deduplication test can be performed, including comparing and deduplicating the raw data itself and comparing and deduplicating the raw data with business system data in a database, where the business system data is previously processed historical data.

[0139] In this scenario example, after deduplication verification, text extraction can be performed on the original data. For the image data in the original data, image recognition can be performed on the image data through OCR technology to extract the text information in the image data. For the text data in the original data, text parsing can be performed on the text data to extract the content in the text data.

[0140] In this scenario, after extracting text from the raw data, the text can be fed into a classification model, which outputs the classification of the raw data. If the raw data matches the target category, key information can be extracted and a brief summary generated. The processed data and the generated summary are then stored in the business system for further analysis and decision support.

[0141] The photovoltaic information analysis method provided in the embodiment of the present application can be executed by a photovoltaic information analysis device. In the embodiment of the present application, the photovoltaic information analysis device provided in the embodiment of the present application is described by taking the photovoltaic information analysis device executing the photovoltaic information analysis method as an example.

[0142] An embodiment of the present application also provides a photovoltaic information analysis device.

[0143] like Figure 3 As shown, the photovoltaic information analysis device includes:

[0144] An acquisition module 310 is used to acquire raw data related to the photovoltaic industry from a data source;

[0145] The extraction module 320 is configured to extract first text information from the text data of the original data; and, if the original data includes image data, extract second text information from the image data;

[0146] An identification module 330 is configured to identify the category of the original data based on the first text information and the second text information;

[0147] The generating module 340 is configured to generate summary information of the original data when the category of the original data is the target category.

[0148] The photovoltaic information analysis device provided in the embodiment of the present application obtains raw data related to the photovoltaic industry from a data source; extracts first text information from the text data of the raw data; and, in the case where the raw data includes image data, extracts second text information from the image data; identifies the category of the raw data based on the first text information and the second text information; and generates summary information of the raw data when the category of the raw data is a target category. The embodiment of the present application can cover various data forms of the photovoltaic industry by extracting key information from multi-source and multi-modal data, reducing the problem of information omission caused by a single data type, thereby being able to accurately identify the category of the raw data, generate summary information after confirming that the raw data belongs to the target category, further refine key content, and provide users with accurate and efficient information output, making it easier to quickly grasp the core dynamics of the photovoltaic industry and improving the efficiency and accuracy of photovoltaic information analysis.

[0149] In some embodiments, the acquisition module 310 is further configured to:

[0150] Perform language identification on raw data;

[0151] In the case where the original data includes data in a language other than the target language, the original data is translated into the target language.

[0152] In some embodiments, the extraction module 320 is further configured to:

[0153] Perform deduplication processing on the original data to delete duplicate data in the original data;

[0154] The deduplicated original data is compared with the historical data in the preset database, and the data that is duplicated between the deduplicated original data and the historical data is deleted.

[0155] In some embodiments, the extraction module 320 is further configured to:

[0156] The text data is input into the large language model, and the data associated with the target content in the text data is extracted by the large language model to obtain the first text information.

[0157] In some embodiments, the extraction module 320 is further configured to:

[0158] In the case where the image data includes text content, converting the text content into second text information by optical character recognition;

[0159] In the case where the image data includes chart information, the chart information is converted into second text information by the large language model.

[0160] In some embodiments, the extraction module 320 is further configured to:

[0161] In the case where the original data includes a file attachment, the file attachment is parsed to obtain the text content of the file attachment;

[0162] The text content of the file attachment is added to the first text message.

[0163] In some embodiments, the identification module 330 is further configured to:

[0164] Fine-tune the large language model to obtain a classification model;

[0165] The first text information and the second text information are input into the classification model to obtain the category of the original data output by the classification model.

[0166] The photovoltaic information analysis device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0167] The photovoltaic information analysis device in the embodiments of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0168] In some embodiments, as Figure 4 As shown, an embodiment of the present application also provides an electronic device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the program is executed by the processor 401, each process of the above-mentioned photovoltaic information analysis method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0169] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0170] An embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the various processes of the above-mentioned photovoltaic information analysis method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0171] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0172] An embodiment of the present application further provides a computer program product, including a computer program, which implements the above-mentioned photovoltaic information analysis method when executed by a processor.

[0173] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0174] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned photovoltaic information analysis method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0175] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0176] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0178] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0179] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0180] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and intent of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A photovoltaic information analysis method, characterized in that: include: Obtain raw data related to the photovoltaic industry from data sources; extracting first text information from the text data of the original data; and, in a case where the original data includes image data, extracting second text information from the image data; identifying a category of the original data according to the first text information and the second text information; When the category of the original data is the target category, summary information of the original data is generated.

2. The method according to claim 1, characterized in that Before extracting the first text information from the text data of the original data, the method includes: performing language recognition on the raw data; In a case where the original data includes data in a language other than the target language, the original data is translated into the target language.

3. The method according to claim 1, characterized in that Before extracting the first text information from the text data of the original data, the method includes: Performing deduplication processing on the original data to delete duplicate data in the original data; The original data after deduplication processing is compared with the historical data in a preset database, and the data in the original data after deduplication processing that is duplicated with the historical data is deleted.

4. The method according to claim 1, wherein The extracting the first text information from the text data of the original data includes: The text data is input into a large language model, and data associated with target content in the text data is extracted using the large language model to obtain first text information.

5. The method according to claim 1, wherein The extracting second text information from the image data comprises: In the case where the image data includes text content, converting the text content into second text information through optical character recognition; In a case where the image data includes chart information, the chart information is converted into second text information by using a large language model.

6. The method according to claim 1, characterized in that The method further comprises: If the original data includes a file attachment, parsing the file attachment to obtain text content of the file attachment; The text content of the file attachment is added to the first text message.

7. The method according to claim 1, characterized in that The identifying the category of the original data according to the first text information and the second text information includes: Fine-tune the large language model to obtain a classification model; The first text information and the second text information are input into a classification model to obtain the category of the original data output by the classification model.

8. A photovoltaic information analysis device, characterized in that: include: The acquisition module is used to obtain raw data related to the photovoltaic industry from data sources; an extraction module, configured to extract first text information from the text data of the original data; and, if the original data includes image data, extract second text information from the image data; an identification module, configured to identify a category of the original data according to the first text information and the second text information; A generating module is used to generate summary information of the original data when the category of the original data is the target category.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images