Photovoltaic system auxiliary installation method and device, electronic equipment and storage medium

Abstract

According to the auxiliary installation method and device of the photovoltaic system, the electronic equipment and the storage medium provided by the embodiment of the invention, the service request information related to the installation of the photovoltaic system input by a user is acquired, and the service request information comprises at least one mode of the text, the image and the audio information; performing intention recognition according to the service request information and an intention recognition model to obtain an intention recognition result; and outputting response information according to the intention recognition result, so that a user can conveniently inquire related information in the installation process of the photovoltaic system, and the installation efficiency of the photovoltaic system can be improved.

Description

Technical Field

[0001] The present application belongs to the technical field of photovoltaic system installation, and in particular, relates to a method, device, electronic equipment and storage medium for auxiliary installation of a photovoltaic system. Background Art

[0002] During the installation process of a photovoltaic system, professional technicians usually query relevant information during the installation process to assist the technicians in the installation. In the prior art, professional technicians usually query relevant books and materials to perform installation based on the books and materials. The efficiency of querying information by querying relevant books and materials is low, which affects the installation efficiency of the photovoltaic system. Summary of the invention

[0003] The embodiments of the present application provide a method, device, electronic device and storage medium for auxiliary installation of a photovoltaic system, which can more conveniently allow users to query relevant information during the installation of the photovoltaic system, thereby improving the installation efficiency of the photovoltaic system.

[0004] In a first aspect, an embodiment of the present application provides a method for auxiliary installation of a photovoltaic system, comprising:

[0005] Acquire service request information about photovoltaic system installation input by a user, wherein the service request information includes at least one modality of text, image, and audio information;

[0006] Performing intent recognition according to the service request information and the intent recognition model to obtain an intent recognition result;

[0007] Output response information according to the intention recognition result.

[0008] In some embodiments, the intention recognition result is service information for a target type of a target device in a photovoltaic system.

[0009] In some embodiments, the service request information includes image information, and outputting response information according to the intention recognition result includes:

[0010] Determining a target network model corresponding to the target type from a plurality of network models;

[0011] Outputting response information based on the target network model and the image information.

[0012] In some embodiments, the target type includes: a detection type, the target network model includes: a detection model, and the outputting response information based on the target network model and the image information includes:

[0013] determining a detection area of ​​the target device from the image information according to the image information and the area detection model;

[0014] Outputting response information according to the detection area and the detection model.

[0015] In some embodiments, the detection model includes: a feature extraction module, a feature processing module, a feature fusion module, a convolutional block attention mechanism module, and a prediction module;

[0016] The feature extraction module is used to extract the first scale feature in the detection area;

[0017] The feature processing module is used to process based on the first scale feature to obtain a multi-scale feature, and output the multi-scale feature to the feature fusion module;

[0018] The feature fusion module is used to perform bidirectional fusion of multi-scale features to obtain multiple multi-scale fusion features, and input each multi-scale fusion feature into the corresponding convolution block attention mechanism module;

[0019] The convolution block attention mechanism module is used to process each multi-scale fusion feature to obtain a target feature corresponding to each multi-scale fusion feature, and input each target feature into the prediction module;

[0020] The prediction module is used to output the detection result.

[0021] In some embodiments, outputting response information according to the intention recognition result includes:

[0022] When the intention recognition result is to obtain installation guide information of the target device, matching the installation guide information in a database based on the device information of the target device;

[0023] Output the installation guide information.

[0024] In some embodiments, outputting response information according to the intention recognition result includes:

[0025] When the intention recognition result is to input a parameter of the target device, acquiring parameter information based on device information of the target device;

[0026] The parameter information is structured, and the structured parameters are sent to the target device to set the parameters of the target device.

[0027] In some embodiments, performing intent recognition according to the service request information and the intent recognition model to obtain an intent recognition result includes:

[0028] In the case where the service request information is one of text, image and audio information, converting the service request information into a vector, and inputting the vector into the intent recognition model to obtain an intent recognition result;

[0029] In the case where the service request information is at least two types of information among text, image and audio information, each information is converted into a vector form, and the vectors corresponding to each information are fused to obtain a fused vector, and the fused vector is input into the intent recognition model to obtain an intent recognition result.

[0030] In a second aspect, an embodiment of the present application provides a photovoltaic system auxiliary installation device, including:

[0031] An acquisition module, used to acquire service request information about photovoltaic system installation input by a user, wherein the service request information includes at least one modality of text, image and audio information;

[0032] An identification module, used to perform intent identification according to the service request information and the intent identification model to obtain an intent identification result;

[0033] An output module is used to output response information according to the intention recognition result.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the computer program.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the methods described above is implemented.

[0036] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the electronic device to execute any of the methods described above.

[0037] A method for assisted installation of a photovoltaic system provided in an embodiment of the present application obtains service request information about the installation of the photovoltaic system input by a user, wherein the service request information includes at least one modality of text, image, and audio information; performs intent recognition based on the service request information and an intent recognition model to obtain an intent recognition result; and outputs response information based on the intent recognition result, which can more conveniently allow users to query relevant information during the installation of the photovoltaic system, thereby improving the installation efficiency of the photovoltaic system.

[0038] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A schematic diagram of a process flow for implementing a photovoltaic system auxiliary installation method provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of a detection model provided in an embodiment of the present application;

[0042] Figure 3 A structural schematic diagram of an assembly detection model is provided for an embodiment of the present application;

[0043] Figure 4 A schematic diagram of the implementation process of the photovoltaic system auxiliary installation method provided in the embodiment of the present application;

[0044] Figure 5 A schematic diagram of the structure of a photovoltaic system auxiliary installation device provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0046] In the drawings, the same reference numerals are used for the same components and the drawings are not drawn to scale. DETAILED DESCRIPTION

[0047] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0048] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0049] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0050] As used in the specification of this application and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrases "if it is determined" or "if it is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce detected" or "in response to detecting" depending on the context.

[0051] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0052] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0053] Before introducing the embodiments of the present application, a brief introduction to the technologies and technical problems in the related art is given.

[0054] During the installation, inspection and maintenance of photovoltaic systems, the current method mainly relies on professional technicians to perform on-site operations and judgments. There are the following problems with professional personnel performing on-site operations and judgments:

[0055] Specialized and complicated operation process: The equipment control of traditional photovoltaic systems, whether local or remote control, requires professionals to perform complex parameter settings through terminals (such as web or app). These operations are extremely unfriendly to customers who lack professional training and are prone to operational errors.

[0056] The information query process is cumbersome: When customers need to query the operation information or attribute information of the equipment, they must perform corresponding operations between various interfaces. In a complex photovoltaic system, this process is cumbersome and time-consuming, reducing the efficiency and experience of users.

[0057] Limitations of existing systems: Manual inspection and automated systems based on preset rules have slow processing speeds, insufficient accuracy, and lack of comprehensive processing capabilities for complex problems. As the scale and complexity of photovoltaic systems increase, user input forms become more diverse (such as text, images, videos, audio, etc.), and existing systems are difficult to efficiently process these multimodal data.

[0058] Insufficient multimodal data processing capabilities: The existing system is unable to effectively process multimodal data, which requires users to switch between different modules and interfaces during operation, increasing operational complexity and time costs, and seriously affecting user experience.

[0059] In addition, during the installation process of the photovoltaic system, professional technicians usually query relevant information during the installation process to assist the technicians in the installation. In the prior art, professional technicians usually query relevant books and materials to perform installation based on the books and materials. The efficiency of querying information by querying relevant books and materials is low, which affects the installation efficiency of the photovoltaic system.

[0060] Based on the problems in the related art, the embodiments of the present application provide a method for auxiliary installation of a photovoltaic system, which can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA) and other electronic devices. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices. The electronic device can be a processor of a smart photovoltaic system or a processor of a photovoltaic system auxiliary installation system.

[0061] Based on the above embodiments, the present application further provides a method for auxiliary installation of a photovoltaic system. Figure 1 A schematic diagram of the implementation process of a photovoltaic system auxiliary installation method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, including:

[0062] Step S101, obtaining service request information about photovoltaic system installation input by a user, where the service request information includes at least one modality of text, image, and audio information.

[0063] In the embodiments of the present application, a photovoltaic system refers to a system that uses solar energy to generate electricity, and generally includes components such as solar panels, inverters, battery energy storage systems (if any), brackets, cables, etc. Photovoltaic system installation refers to assembling and installing these components to a designated location so that they can generate electricity and supply power to a grid or load.

[0064] In an embodiment of the present application, the service request information may be at least one of text (such as text description, question, etc.), image (such as on-site photos, equipment pictures, etc.) or audio (such as voice description, voice command, etc.).

[0065] In the embodiment of the present application, the image may include: a picture or a video.

[0066] In an embodiment of the present application, the system can receive service request information input by the user through a user interface (such as a web page, APP, chat robot, etc.).

[0067] Step S102, performing intent recognition based on the service request information and the intent recognition model to obtain an intent recognition result.

[0068] In the embodiment of the present application, the intent recognition model is a machine learning model for analyzing the information input by the user and identifying the user's intention or needs. The intent recognition model can be an LLM (large language model), and the LLM (large language model) can be used to recognize the intent of the user input. For example, the intent recognition model can recognize that the user wants to understand the installation guide, parameter settings, defect detection, etc. of the photovoltaic system.

[0069] Step S103: output response information according to the intention recognition result.

[0070] In the embodiment of the present application, the response information may be feedback or answers provided by the system to the user. For example, if the user wants to know the installation steps of the photovoltaic system, the response information may be a detailed installation guide or video tutorial.

[0071] In the embodiments of the present application, tasks can be automatically assigned and processed through intent recognition, and response information can be automatically determined, thereby achieving automation and intelligence of complex tasks, reducing the user's operating steps and time cost, and improving the system's response speed and accuracy.

[0072] In the embodiment of the present application, the response information can be displayed to the user through the user interface. The response information can be in various forms such as text, image, audio or video.

[0073] The method provided in the embodiment of the present application obtains service request information input by the user regarding the installation of the photovoltaic system, the service request information including at least one modality of text, image, and audio information; performs intent recognition based on the service request information and the intent recognition model to obtain the intent recognition result; and outputs response information based on the intent recognition result, which can more conveniently allow the user to query relevant information during the installation of the photovoltaic system, thereby improving the installation efficiency of the photovoltaic system.

[0074] The method provided in the embodiment of the present application can comprehensively process multimodal data (such as text, images, video, audio, etc.). Users do not need to switch between different modules. The operation is simple and can improve the user experience.

[0075] In some embodiments, the intention recognition result is service information for a target type of a target device in the photovoltaic system.

[0076] Target devices refer to the specific photovoltaic devices that users want to understand or operate. These devices may be any component in the photovoltaic system, each of which has its own specific functions, performance parameters and operation procedures. Target type service information may include: service information such as installation guides, installation recommendations, performance parameter queries, and testing.

[0077] In some embodiments, the service request information includes image information. Step S103 outputs response information according to the intention recognition result, including:

[0078] Step S1031, determining a target network model corresponding to the target type from multiple network models.

[0079] In the embodiment of the present application, multiple network models may be preset in the system, each of which is specifically used to process or analyze a specific type of service information. In this scenario, the system will select a network model that best matches the target type from multiple network models as the target network model based on the service information of the target type determined in the intent recognition result.

[0080] Exemplarily, if the service information of the target type is defect detection, the target model is a target network model corresponding to the defect detection.

[0081] Step S1032: output response information based on the target network model and image information.

[0082] In the embodiment of the present application, when the target network model is determined, the image information provided by the user is input into the model, and the target network model analyzes and processes the image information to extract features or information related to the user's intention. Based on these features or information, the system can generate corresponding response information. The form and content of the response information depends on the user's intention and the function of the target network model.

[0083] For example, during the installation of a photovoltaic system, users can upload photos of the installed equipment. The system can automatically identify the type of equipment and whether there are any defects, and then give the corresponding defects and provide treatment suggestions.

[0084] In some embodiments, the target type includes: detection type, and the detection type may specifically include wiring detection, defect detection, assembly detection, etc.

[0085] In the embodiment of the present application, the target network model includes: a detection model, which outputs response information based on the target network model and image information, including:

[0086] Step S1, determining a detection area of ​​a target device from image information according to image information and an area detection model.

[0087] In an embodiment of the present application, the image information may be a photo or image data containing the target device. Region Detection Model This is a pre-trained model used to identify the location of a specific object in an image. In this scenario, the region detection model is used to identify and locate the target device (such as a solar panel, inverter, etc.) in the image.

[0088] In the embodiment of the present application, after the region detection model processes the image information, it will output one or more bounding boxes, which mark the location of the target device in the image. The area surrounded by these bounding boxes is the detection area. The image information can be input into the region detection model, so that the detection area of ​​the target device can be obtained. The region detection model can detect the information of the target device and determine the detection area.

[0089] Step S2: outputting response information according to the detection area and the detection model.

[0090] In the embodiment of the present application, the detection model is a model trained for the detection type of the target device, which is used to analyze the image information in the detection area and output response information related to the user's intention. The detection model may be a model based on deep learning, which can recognize specific features in the image and generate response information based on these features.

[0091] In the embodiment of the present application, the detection area can be input into the detection model, so as to obtain response information. The response information may include detection results, such as wiring detection results, hardware defect detection results, assembly detection results, etc.

[0092] When the detection type is defect detection, the detection model is a defect detection model. At this time, after obtaining the image information, the area detection will be triggered. You can get a regional detection model from the object storage service, and use the regional detection model to identify the detection area of ​​the specific component. The detection area is the hardware component area, and then use this specific component to obtain the detection model corresponding to the specific component for detection. The detection results may include: images after the defect detection results are framed and marked with errors, and the defect detection results may include: defect coordinates. In the defect detection during the photovoltaic installation process, if there are no defect coordinates, then the hardware components are normal and without defects, and the information that the components are successfully installed can be output; if there are defects, the large language model can organize the text and error correction pictures to form an answer feedback to the user.

[0093] When the detection type is assembly detection, the detection model is the assembly detection model. At this time, after obtaining the image information, the area detection will be triggered. You can obtain a regional detection model from the object storage service, and use the regional detection model to identify the detection area of ​​the specific device. The detection area is the wiring detection area, and then obtain the assembly detection model corresponding to the device for detection. The detection results may include: wiring errors or no wiring errors, and the response information is output based on the detection results. In the assembly detection during the photovoltaic installation process, if there is an error in the wiring after the detection, the wiring error information can be output, and the specific wrong wiring can be identified. If there is no error in the wiring, the large language model can output feedback information of successful wiring.

[0094] When the detection type is assembly detection, the detection area is the assembly area of ​​the equipment. The equipment is then tested using the assembly detection model corresponding to the equipment to obtain the assembly result. The assembly result can be fed back through the large language model. If the assembly is incorrect, the assembly result (text + picture) will be fed back through the large language model as a reminder.

[0095] In some embodiments, the detection model can be a defect detection model and a wiring detection model. The detection model includes: a feature extraction module, a feature processing module, a feature fusion module, a convolutional block attention mechanism module and a prediction module; the feature extraction module is used to extract the first scale feature in the detection area; the feature processing module is used to process based on the first scale feature to obtain multi-scale features, and output the multi-scale features to the feature fusion module; the feature fusion module is used to bidirectionally fuse the multi-scale features to obtain multiple multi-scale fusion features, and input each multi-scale fusion feature into the corresponding convolutional block attention mechanism module; the convolutional block attention mechanism module is used to process each multi-scale fusion feature to obtain the target feature corresponding to each multi-scale fusion feature, and input each target feature into the prediction module; the prediction module is used to output the detection result. The convolutional attention mechanism module can be: a CBAM attention mechanism model.

[0096] In the embodiment of the present application, Figure 2 A schematic diagram of a detection model provided in an embodiment of the present application, such as Figure 2As shown, the feature fusion module is a bidirectional feature pyramid structure, and the feature processing module includes: an input layer, a first convolution layer, a second convolution layer, a multi-head attention layer, and a third convolution layer. The first input layer is used to input a first scale feature, and output the first scale feature to the feature fusion module and the first convolution layer. The first convolution layer is used to perform convolution processing on the first size feature to obtain a second scale feature, and output the second scale feature to the second convolution layer and the feature fusion module. The second convolution layer is used to perform convolution processing on the second scale feature to obtain a third scale feature and output the third scale feature to the multi-head attention layer. The multi-head attention layer is used to determine based on the third scale feature Each head has an attention weight and output for each element, and the outputs of all heads are spliced ​​to obtain the processed third-scale features, and the third-scale features are output to the third convolutional layer and the feature fusion module. The third convolutional layer is used to perform convolution processing on the third-scale features to obtain the fourth-scale features, and the fourth-scale features are output to the feature fusion module. The bidirectional feature pyramid structure outputs four scale features to the corresponding convolution block attention mechanism modules respectively, where the scale of the first scale feature is greater than the scale of the second scale feature, the scale of the second scale feature is greater than the scale of the third scale feature, and the scale of the third scale feature is greater than the scale of the fourth scale feature.

[0097] In the embodiment of the present application, the first scale feature may be a large scale feature, the second scale feature may be a medium scale feature, the third scale feature may be a small scale feature, and the fourth scale feature may be an extremely small scale feature. The input layer may be a feature extraction network with CSPDarknet 53 as the basic structure.

[0098] In the embodiment of the present application, when targeting the defect detection model, the multi-head attention layer can capture the relationship between the defect and the background and enhance the distinctiveness of the features. A bidirectional feature pyramid structure is used to fuse large and small scale features to ensure that each scale has feature information of different depths, rich deep semantic information, and rich shallow detail information. The convolutional block attention mechanism module is placed in the previous layer of the prediction module, which also plays a role in suppressing background noise and highlighting the target in the prediction of each scale. When the detection model is a wiring detection model, the multi-head attention layer can capture global information while paying attention to surrounding information, which is also conducive to wiring detection.

[0099] In the embodiment of the present application, the multi-head attention layer can calculate the attention weight of each feature, and the attention score can be calculated by the attention weight. Among them, the attention weight can be calculated by the following formula:

[0100] Q=XW Q ;

[0101] K=XW K ;

[0102] V=XWV ;

[0103] Among them, X∈R^(n×d) is the input feature, where n is the sequence length and d is the feature dimension; W Q ,W V ,W K ∈R^(n×d_k) is the learned weight parameter, and d_k is the feature dimension of each head.

[0104] The attention score can be calculated by the following formula:

[0105]

[0106] Among them, QK T : Calculate the dot product between the query and the key to generate a correlation matrix; is the scaling factor; softmax(.) performs a softmax operation on the correlation matrix to obtain the attention weight, and P is the position code.

[0107] In the embodiment of the present application, after calculating the attention score, the multi-head attention can be calculated, and the calculation method can be expressed as:

[0108]

[0109] in, is the weight matrix of the i-th head.

[0110] After determining the multi-head attention, the outputs can be merged.

[0111] MHSA(X)=Concat(head 1 ,...,head h )W 0 ;

[0112] in,

[0113] In an embodiment of the present application, the multi-head attention layer is placed after the small-scale features of the feature pyramid, and the feature maps of different channels are flattened and used as feature vectors, which can capture the relationship between different pixels, thereby enabling the model to focus on the spatial relationship between regions, and in the channel dimension, it can focus on the relationship between different features of the target.

[0114] In some embodiments, if the detection model is an assembly detection model, the detection model has obvious differences in the characteristics of various components and wiring areas, and is large in size. Therefore, the computational efficiency is taken into consideration, the computational complexity is reduced, and the extremely small scale detection branch and the multi-head attention layer are removed. Figure 3 A structural diagram of an assembly detection model is provided for an embodiment of the present application, such as Figure 3 shown.

[0115] In the embodiment of the present application, the loss function of the detection model includes: classification loss and prediction loss. The classification loss can be expressed as:

[0116]

[0117] The prediction loss is: the loss that measures the difference between the predicted box and the true box, the loss function EIoU loss:

[0118]

[0119] Where P is the Euclidean distance of the bright spot; b is the coordinate of the center point of the prediction box, b gt The coordinates of the center point of the real frame. c is the width of the bounding box, h c is the bounding box height.

[0120] The total loss includes:

[0121] total_loss = λ cls cls_loss+λ box_loss box_loss.

[0122] According to the method provided in the embodiment of the present application, the user uploads images or videos, and the system automatically performs defect and wiring detection and provides multi-modal output results, thereby reducing dependence on professional technicians and improving the accuracy and efficiency of detection.

[0123] In some embodiments, step S103, outputting response information according to the intention recognition result, includes:

[0124] Step S11, when the intention recognition result is to obtain the installation guide information of the target device, the installation guide information is matched in the database based on the device information of the target device.

[0125] In the embodiment of the present application, after identifying the intent, the system needs to extract the specific information of the target device from the user's query. This may include the model, brand, serial number, etc. of the device, depending on the requirements of the installation guide information retrieval. After extracting the target device information, the system will search for installation guide information that matches the device information in a pre-established database. This database may contain installation guides for multiple devices, each of which is associated with specific device information.

[0126] In the embodiment of the present application, the system will traverse the stored installation guide information in the database to find the entry that fully matches or is closest to the target device information. The matching process may involve exact matching, fuzzy matching, or recommendation based on a certain algorithm for the device information.

[0127] In the embodiment of the present application, the most suitable installation guide is matched by means of vector search or the like.

[0128] Step S12: output installation guide information.

[0129] In the implementation of this application, when matching installation guide information is found, the system will extract it and present it to the user through an appropriate output method. This may include displaying it on a web page, sending it via email, downloading it as a PDF file, etc. In some embodiments, the installation guide has a URL link, and multimodal data (pictures, videos) in the object storage service can be obtained through the URL, and the guide text results are then handed over to the large language model for organization and return, and finally the text, pictures, and videos are rendered and presented to the user through the front end.

[0130] In some embodiments, step S103 may also be implemented by the following steps:

[0131] Step S21, when the intention recognition result is to input parameters of the target device, obtain parameter information based on the device information of the target device.

[0132] In an embodiment of the present application, after identifying the intent, the system will extract specific information of the target device from the user's query or input, such as the device model, brand, serial number or network address, etc. This information will be used to locate the target device in subsequent steps.

[0133] In the embodiment of the present application, the device information of the target device may be matched with the parameter information, where the parameter information may include: configuration options, working modes, network settings, etc. of the device.

[0134] Step S22, structuring the parameter information and sending the structured parameters to the target device to set the parameters of the target device.

[0135] In the embodiment of the present application, the system structures the parameter information provided by the user, that is, converts it into a formatted data form so that it can be correctly parsed and recognized by the target device. This usually involves clearly marking the parameter name, value and data type.

[0136] In the embodiment of the present application, after completing the structured processing of the parameter information, the system will send the structured parameters to the target device through an appropriate communication protocol (such as TCP / IP, HTTP, MQTT, etc.). After the target device completes the parameter setting, it may send feedback or confirmation information to the system, indicating that the parameters have been successfully set or modified. The system can provide the user with status updates or operation results based on this information.

[0137] In some embodiments, step S102 may be implemented by the following steps:

[0138] Step S1021, when the service request information is one of text, image and audio information, the service request information is converted into a vector, and the vector is input into the intent recognition model to obtain an intent recognition result.

[0139] In the embodiment of the present application, when the service request information is text, the text is first converted into a vector using natural language processing (NLP) technology. This is usually achieved through techniques such as word embedding (such as Word2Vec, GloVe) or sentence embedding (such as BERT, GPT), which can convert text data into vector representations in high-dimensional space. The obtained text vector is input into the intent recognition model, which may be a classifier based on deep learning (such as LSTM, GRU, Transformer) for identifying intent in text.

[0140] When the service request information is image information, image processing techniques such as convolutional neural networks (CNN) are used to convert the image into a vector representation. This usually involves image preprocessing (such as scaling, cropping, normalization), feature extraction (such as edge detection, texture analysis), and vector generation (such as output through a fully connected layer). The image vector is input into the intent recognition model to identify the intent or content in the image.

[0141] When the service request information is audio information, audio processing techniques (such as Fourier transform and Mel-frequency cepstral coefficients MFCC) can be used to convert the audio signal into a vector representation. These techniques can extract features in the audio signal, such as frequency, amplitude, and timbre. The audio vector is input into the intent recognition model to identify the intent or voice content in the audio.

[0142] Step S1022, when the service request information is at least two types of information among text, image and audio information, each information is converted into a vector form, and the vectors corresponding to each information are fused to obtain a fused vector, and the fused vector is input into the intent recognition model to obtain an intent recognition result.

[0143] In an embodiment of the present application, when the service request information contains at least two of text, images and audio, each type of information is first converted into a vector form using the above-mentioned technology. The vectors corresponding to each type of information are fused. Fusion methods may include simple splicing, weighted averaging, attention mechanisms, and the like. The splicing method directly connects vectors of different information types together to form a longer vector; the weighted averaging method performs a weighted summation of each vector according to the importance of the information; and the attention mechanism dynamically adjusts the weight of each vector by calculating the correlation between different vectors. The fused vector is input into the intent recognition model. This model may be specially designed to process fused vectors, and can recognize the intent expressed by multiple types of information.

[0144] In some embodiments, outputting response information according to the intent recognition result includes:

[0145] When the intention recognition result is to conduct intelligent question and answer and information query, the result of the question and answer can be output.

[0146] In the embodiment of the present application, users can directly obtain questions and answers and information in professional fields through natural language or multimodal input, which improves the convenience and intelligence of the query.

[0147] In some embodiments, multimodal data related to the photovoltaic system can be collected and prepared, including text data, image data, video data, and audio data. These data cover various aspects such as equipment control, information query, defect detection, and wiring detection. Use the collected multimodal data to train and fine-tune the large language model and the multimodal large model. Ensure that the model can accurately handle tasks related to the photovoltaic system and improve processing efficiency and accuracy. During the training process, use data from professional fields for reinforcement learning and fine-tuning to improve the professionalism and reliability of the model in the photovoltaic field.

[0148] Based on the aforementioned embodiments, the embodiments of the present application further provide a method for assisted installation of a photovoltaic system. By integrating a large language model, a visual language model, and an intelligent photovoltaic system with speech recognition, intelligent guidance and error correction can be performed during the photovoltaic installation process, thereby reducing the error rate, speeding up the installation process, increasing the installation success rate, and reducing costs.

[0149] Figure 4 A schematic diagram of the implementation flow of the photovoltaic system auxiliary installation method provided in the embodiment of the present application is shown in FIG. Figure 4 As shown, including:

[0150] Users input into the smart photovoltaic system in the form of pictures, text, and voice. The smart photovoltaic system will use the visual language model to understand the picture to form an embedding vector. The voice or text will also be transformed into an embedding vector through the embedding model. The two will be fused and input processed through the large language model to output a category, which is the intention recognition result.

[0151] The user inputs the model in the form of a combination of pictures + text (voice), etc. The smart photovoltaic system recognizes the intention to obtain the installation guide, understands the components in the picture through image understanding tasks, and matches the most suitable installation guide through the output component category plus additional text through vector search and other methods. It is returned to the smart photovoltaic system, which obtains multimodal data (pictures, videos) in the object storage service through the URL link in the guide, and then gives the guide text results to the large language model for organization and return, and finally renders the text, pictures, and videos to the user through the front end.

[0152] After the installation guide is presented to the user, the user will install the system. During the installation process, the input is still in the form of picture + text (voice). After being fused in the form of embedded vectors, the intention can be recognized through the large language model. When the hardware defect detection task is identified, the region detection will be triggered. A region detection model is obtained through the object storage service. The specific hardware component region is identified through the region detection model. Then, the hardware detection model of the specific component in the object storage service is obtained through this specific component for detection. The detection result is the defect coordinates and the image after the error is marked in the box. The defect coordinates are then judged by the smart photovoltaic system. If there are no defect coordinates, the hardware component is normal and has no defects, and the component is installed successfully. If there are defects, the large language model organizes the text and error correction pictures to form an answer and feedback to the user. When the detection task is a device assembly task, the specific assembly area is also obtained through region detection, and then the specific component assembly detection model is used for detection. If the assembly is incorrect, the assembly result (text + picture) will be fed back through the large language model to remind. The intention recognition process for the wiring assembly task is the same as above. The specific wiring area is then obtained, and then the detection model is used for detection through the specific area assembly. If a wiring error is identified, the specific incorrect wiring will be marked, and the assembly result (text + picture) will be fed back through the large language model.

[0153] If the error correction process is correct through the intelligent photovoltaic system reminder, and the system statistics show that all areas have been installed, it means that the installation is complete.

[0154] After the installation is completed, parameters can be sent and quality checked. They are sent through the smart photovoltaic system. After the intention is recognized as parameter setting, the parameters are structured through the large language model, and the command interface is called and sent to the specific device. The device self-checks, after the intention is recognized, and self-checks specific devices and modules. The parameter setting results and device self-check results are organized by the large language model and returned to the user.

[0155] If the above process is normal, the installation is successful. Otherwise, follow the prompts of the smart photovoltaic system until the installation is successful.

[0156] The method provided in the embodiment of the present application integrates a multimodal large model, which can process data in various forms including text, images, videos, audio, etc., and supports users to interact with the system through multimodal input. The system's ability to process various forms of data has been greatly improved. Users can obtain required information and services through any one or more forms of input, which improves the convenience and flexibility of operation. The user input is recognized by LLM (large language model), corresponding prompts are generated according to different intentions, and the input and prompts are passed to the next processing module. The system can accurately identify user intentions, automatically assign tasks and process them, realize the automation and intelligence of complex tasks, reduce the user's operation steps and time costs, and improve the response speed and accuracy of the system. Each module of the system (such as equipment control, information query, defect detection, wiring detection, etc.) realizes intelligent collaboration and integrated processing through a multimodal large model, unifying the system operation process. It solves the problems of independent functional modules and complex operations in the existing system, realizes the collaborative processing and integrated management of various functional modules, and improves the overall efficiency and user experience of the system. Using a multimodal large model for tasks such as target detection, text recognition, and image and video understanding can automatically detect hardware component defects and wiring conditions. Users upload images or videos, and the system automatically performs defect and wiring inspections and provides multi-modal output results, reducing reliance on professional technicians and improving the accuracy and efficiency of inspections. The system can obtain professional field data from a variety of data sources (such as MySQL, Milvus, Elasticsearch, etc.) based on the prompts generated by LLM, and conduct intelligent Q&A and information query. Users can directly obtain professional field Q&A and information through natural language or multi-modal input, which improves the convenience and intelligence of queries.

[0157] Based on the foregoing embodiments, the embodiments of the present application provide a photovoltaic system auxiliary installation device. The modules included in the photovoltaic system auxiliary installation device and the units included in the modules can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, Microprocessor Unit), a digital signal processor (DSP, Digital Signal Processing) or a field programmable gate array (FPGA, Field Programmable Gate Array), etc. Figure 5 A schematic diagram of a photovoltaic system auxiliary installation device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the photovoltaic system auxiliary installation device 500 includes:

[0158] An acquisition module 501 is used to acquire service request information about photovoltaic system installation input by a user, wherein the service request information includes at least one modality of text, image and audio information;

[0159] An identification module 502 is used to perform intent identification according to the service request information and the intent identification model to obtain an intent identification result;

[0160] The output module 503 is used to output response information according to the intention recognition result.

[0161] In some embodiments, the intention recognition result is service information for a target type of a target device in a photovoltaic system.

[0162] In some embodiments, the service request information includes image information, and the output module includes:

[0163] A first determining unit, configured to determine a target network model corresponding to the target type from a plurality of network models;

[0164] The first output unit is used to output response information based on the target network model and the image information.

[0165] In some embodiments, the target type includes: a detection type, the target network model includes: a detection model, and the first output unit includes:

[0166] A first determining subunit, configured to determine a detection area of ​​the target device from the image information according to the image information and an area detection model;

[0167] An output subunit is used to output response information according to the detection area and the detection model.

[0168] In some embodiments, the detection model includes: a feature extraction module, a feature processing module, a feature fusion module, a convolutional block attention mechanism module, and a prediction module;

[0169] The feature extraction module is used to extract the first scale feature in the detection area;

[0170] The feature processing module is used to process based on the first scale feature to obtain a multi-scale feature, and output the multi-scale feature to the feature fusion module;

[0171] The feature fusion module is used to perform bidirectional fusion of multi-scale features to obtain multiple multi-scale fusion features, and input each multi-scale fusion feature into the corresponding convolution block attention mechanism module;

[0172] The convolution block attention mechanism module is used to process each multi-scale fusion feature to obtain a target feature corresponding to each multi-scale fusion feature, and input each target feature into the prediction module;

[0173] The prediction module is used to output the detection result.

[0174] In some embodiments, the output module includes:

[0175] A matching unit, configured to match the installation guide information in a database based on the device information of the target device when the intention recognition result is to obtain the installation guide information of the target device;

[0176] The second output unit is used to output the installation guide information.

[0177] In some embodiments, the output module includes:

[0178] a parameter determination unit, configured to acquire parameter information based on device information of the target device when the intention recognition result is to input a parameter of the target device;

[0179] The sending unit is used to structure the parameter information and send the structured parameters to the target device to set the parameters of the target device.

[0180] In some embodiments, the identification module includes:

[0181] A first recognition unit, configured to convert the service request information into a vector when the service request information is one of text, image and audio information, and input the vector into the intention recognition model to obtain an intention recognition result;

[0182] The second recognition unit is used to convert each information into a vector form when the service request information is at least two types of information among text, image and audio information, and fuse the vectors corresponding to each information to obtain a fused vector, and input the fused vector into the intent recognition model to obtain an intent recognition result.

[0183] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device 3 of this embodiment may include: at least one processor 30 ( Figure 6Only one processor 30 is shown in the figure), a memory 31, and a computer program 32 stored in the memory 31 and executable on at least one processor 30. When the processor 30 executes the computer program 32, the steps in any of the above-mentioned method embodiments are implemented; or, when the processor 30 executes the computer program 32, the functions of the modules / units in the above-mentioned system embodiments are implemented.

[0184] Exemplarily, the computer program 32 may be divided into one or more modules / units, one or more modules / units are stored in the memory 31, and are executed by the processor 30 to complete the present application. One or more modules / units may be a series of computer program 32 instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 32 in the electronic device 3.

[0185] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program 32. When the computer program 32 is executed by the processor 30, the steps in the above-mentioned method embodiments can be implemented.

[0186] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0187] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. According to this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program 32, and the computer program 32 can be stored in a computer-readable storage medium, and the computer program 32 can implement the steps of the above-mentioned various method embodiments when executed by the processor 30. Among them, the computer program 32 includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the terminal, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0188] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0189] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0190] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0191] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for auxiliary installation of a photovoltaic system, characterized in that: include: Acquire service request information about photovoltaic system installation input by a user, wherein the service request information includes at least one modality of text, image, and audio information; Performing intent recognition according to the service request information and the intent recognition model to obtain an intent recognition result; Output response information according to the intention recognition result.

2. The method according to claim 1, characterized in that The intention recognition result is service information for a target type of a target device in the photovoltaic system.

3. The method according to claim 2, characterized in that The service request information includes image information, and the response information is output according to the intention recognition result, including: Determining a target network model corresponding to the target type from a plurality of network models; Outputting response information based on the target network model and the image information.

4. The method according to claim 3, characterized in that The target type includes: a detection type, the target network model includes: a detection model, and the outputting response information based on the target network model and the image information includes: determining a detection area of ​​the target device from the image information according to the image information and the area detection model; Outputting response information according to the detection area and the detection model.

5. The method according to claim 4, characterized in that The detection model includes: a feature extraction module, a feature processing module, a feature fusion module, a convolution block attention mechanism module and a prediction module; The feature extraction module is used to extract the first scale feature in the detection area; The feature processing module is used to process based on the first scale feature to obtain a multi-scale feature, and output the multi-scale feature to the feature fusion module; The feature fusion module is used to perform bidirectional fusion of multi-scale features to obtain multiple multi-scale fusion features, and input each multi-scale fusion feature into the corresponding convolution block attention mechanism module; The convolution block attention mechanism module is used to process each multi-scale fusion feature to obtain a target feature corresponding to each multi-scale fusion feature, and input each target feature into the prediction module; The prediction module is used to output the detection result.

6. The method according to claim 2, characterized in that: The outputting response information according to the intention recognition result includes: When the intention recognition result is to obtain installation guide information of the target device, matching the installation guide information in a database based on the device information of the target device; Output the installation guide information.

7. The method according to claim 2, characterized in that The outputting response information according to the intention recognition result includes: When the intention recognition result is to input a parameter of the target device, acquiring parameter information based on device information of the target device; The parameter information is structured, and the structured parameters are sent to the target device to set the parameters of the target device.

8. The method according to any one of claims 1 to 7, characterized in that: Intent recognition is performed according to the service request information and the intent recognition model to obtain an intent recognition result, including: In the case where the service request information is one of text, image and audio information, converting the service request information into a vector, and inputting the vector into the intent recognition model to obtain an intent recognition result; In the case where the service request information is at least two types of information among text, image and audio information, each information is converted into a vector form, and the vectors corresponding to each information are fused to obtain a fused vector, and the fused vector is input into the intent recognition model to obtain an intent recognition result.

9. A photovoltaic system auxiliary installation device, characterized in that: include: An acquisition module, used to acquire service request information about photovoltaic system installation input by a user, wherein the service request information includes at least one modality of text, image and audio information; An identification module, used to perform intent identification according to the service request information and the intent identification model to obtain an intent identification result; An output module is used to output response information according to the intention recognition result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images