Photovoltaic station fault maintenance positioning method and system based on digital twinning

Abstract

The invention provides a photovoltaic station fault maintenance positioning method and system based on digital twinning, and the method comprises the steps: collecting an image of a fault photovoltaic string through an image collection device after the fault of the photovoltaic string is recognized, and transmitting the image to a three-dimensional digital twinning model of a photovoltaic station; carrying out feature extraction on the collected image, and carrying out matching comparison with a photovoltaic panel image feature library to determine a photovoltaic panel with a fault; the image acquisition device identifies the maintainer to obtain a two-dimensional coordinate of the maintainer in the image, converts the two-dimensional coordinate into a three-dimensional coordinate, and transmits the three-dimensional coordinate to the three-dimensional digital twin model; and the three-dimensional digital twin model interacts the position information of the maintainer in real time with the terminal equipment of the maintainer, plans a path for the maintainer to arrive at the position of the fault photovoltaic panel, and guides the path. The method has the advantages that the reliability and timeliness of the photovoltaic field station from fault discovery to maintenance processing are guaranteed, and the operation and maintenance management level of the field station is effectively improved.

Description

Technical Field

[0001] The present application belongs to the technical field of photovoltaic station operation and maintenance, and specifically relates to a photovoltaic station fault maintenance and positioning method and system based on digital twins. Background Art

[0002] With the booming photovoltaic power generation industry, the scale of photovoltaic stations continues to expand and their structures become increasingly complex. It is increasingly important to ensure their stable and efficient operation and to promptly and accurately repair faulty photovoltaic components. Photovoltaic power stations use operating parameters to diagnose photovoltaic panel faults. The current and voltage generally collected are data based on photovoltaic panel strings. The minimum fault unit obtained by fault diagnosis is the photovoltaic panel string, which is difficult to accurately identify a specific photovoltaic panel. Further accurate diagnosis requires maintenance personnel to conduct on-site inspections. In some photovoltaic stations, the satellite signal is weak or there is interference, resulting in satellite positioning accuracy that is difficult to meet the needs of guiding maintenance personnel to specific photovoltaic panels. In recent years, the height of photovoltaic panel brackets has increased, and the height of photovoltaic panels exceeds the field of view of maintenance personnel. Maintenance personnel reach the fault area through planned paths, and it is difficult to confirm the surface status of photovoltaic panels directly below the photovoltaic panels. The final accurate diagnosis is inefficient and wastes manpower and material resources. Summary of the invention

[0003] The purpose of this application is to use digital twin technology, based on the determination of string-level faults, based on a three-dimensional digital twin model, using white light cameras and infrared cameras to capture images of faulty strings, and using image recognition technology to locate the faulty photovoltaic panels in the faulty strings, to solve the problem of rapid fault location at the accuracy level of a single photovoltaic panel. Maintenance personnel do not rely on satellite positioning. Through image recognition technology, maintenance personnel are included in the digital twin model, achieving real-time interaction between the location information of maintenance personnel and the location information of personnel in the virtual model, and guiding maintenance personnel to quickly and accurately reach the faulty photovoltaic panel to perform fault repairs.

[0004] In order to achieve the above objectives, this application proposes a photovoltaic station fault inspection and positioning method based on digital twins, including:

[0005] When a photovoltaic string failure is identified, the image acquisition device collects images of the faulty photovoltaic string and sends them to the three-dimensional digital twin model of the photovoltaic station;

[0006] Extract features from the collected images and compare them with the photovoltaic panel image feature library to identify the faulty photovoltaic panel;

[0007] The image acquisition device identifies the maintenance personnel and obtains the two-dimensional coordinates of the maintenance personnel in the image, converts the two-dimensional coordinates of the maintenance personnel into three-dimensional coordinates, and transmits them to the three-dimensional digital twin model;

[0008] The three-dimensional digital twin model interacts with the maintenance personnel’s terminal devices in real time to monitor the maintenance personnel’s location information, plan a path for the maintenance personnel to reach the location of the faulty photovoltaic panel, and provide guidance.

[0009] As an improvement of the above method, the method further includes:

[0010] Use image acquisition equipment to collect data from photovoltaic stations;

[0011] After preprocessing the collected images, three-dimensional modeling software is used to generate a three-dimensional digital twin model of the photovoltaic station.

[0012] As an improvement of the above method, the image acquisition device acquires images of the faulty photovoltaic string, including white light images and infrared images.

[0013] As an improvement of the above method, the step of determining a failed photovoltaic panel includes:

[0014] Performing grayscale processing, histogram equalization processing and denoising filtering processing on the white light image; extracting shape features, texture features and color features of the photovoltaic panel from the white light image as image features;

[0015] Performing temperature correction and thermal imaging enhancement processing on the infrared image; adopting a multi-scale analysis method based on wavelet transform on the infrared image to obtain the macroscopic temperature gradient distribution of the hot spot and the microscopic temperature fluctuation inside the hot spot as image features;

[0016] The extracted image features are matched and compared with the photovoltaic panel image feature library, and the similarity calculation method is used to determine the faulty photovoltaic panel.

[0017] As an improvement of the above method, the image acquisition device identifies the maintenance personnel to obtain the two-dimensional coordinates of the maintenance personnel in the image, including:

[0018] Using the trained object detection model, the maintenance personnel in the image are identified to obtain the bounding box of the maintenance personnel in the image;

[0019] Calculate the two-dimensional coordinates (u, v) of the maintenance personnel in the image according to the coordinates of the bounding box:

[0020]

[0021] Among them, x pmin 、x pmax ,y pmin ,y pmax They are the minimum x-axis value, maximum x-axis value, minimum y-axis value, and maximum y-axis value of the maintenance personnel boundary box respectively.

[0022] As an improvement of the above method, the step of converting the two-dimensional coordinates of the maintenance personnel into three-dimensional coordinates includes:

[0023] Create a 3×4 dimensional perspective transformation matrix M:

[0024]

[0025] Select n, n ≥ 4 reference points, and substitute the coordinates of the reference points into the following formula to calculate the element values ​​of the perspective transformation matrix M of the two-dimensional coordinate solution:

[0026]

[0027] Among them, (u i , v i ) represents the two-dimensional coordinates of the i-th reference point; (X i , Y i , Z i ) represents the three-dimensional coordinates of the i-th reference point;

[0028] The three-dimensional coordinates of the maintenance personnel are calculated by the following formula:

[0029]

[0030] Among them, (X, Y, Z) represents the three-dimensional coordinates of the maintenance personnel.

[0031] As an improvement of the above method, the reference point screening method includes:

[0032] Calculate the spatial distance between each reference point and other reference points:

[0033]

[0034] Among them, d ij represents the spatial distance between the i-th reference point and the j-th reference point;

[0035] Calculate the distance distribution entropy between each reference point and other reference points:

[0036]

[0037] Among them, H i represents the distance distribution entropy of the i-th reference point; the spatial distance is evenly divided into several intervals, f ik Indicates the distance d ij The frequency of falling into the kth interval;

[0038] Obtain n reference points with the largest distance distribution entropy as the reference points for final screening.

[0039] As an improvement of the above method, the converting of the two-dimensional coordinates of the maintenance personnel into three-dimensional coordinates further includes:

[0040] The initial value of the perspective transformation matrix M0 is calculated based on the reference point n with the largest distance distribution entropy in the image, and the initial value of the three-dimensional position of the maintenance personnel (X p0 , Y p0 , Z p0 );

[0041] Calculate the distance between other reference points and the initial value of the maintenance personnel's three-dimensional position:

[0042]

[0043] Among them, d ip0 Indicates the distance between the i-th reference point and the initial value of the maintenance personnel's three-dimensional position;

[0044] Calculate the selection preference for each reference point:

[0045]

[0046] Among them, S i0 represents the selection tendency of the i-th reference point; α represents the weight coefficient;

[0047] Select n reference points with the highest selection tendency and recalculate the perspective transformation matrix M1, and calculate the three-dimensional coordinates of the maintenance personnel through the perspective transformation matrix M1;

[0048] Repeat the above process for a set number of times to improve the accuracy of the maintenance personnel's three-dimensional coordinates.

[0049] The present application also provides a photovoltaic station fault inspection and positioning system based on digital twins, which is implemented based on the above method, and the system includes:

[0050] A module for collecting images of faulty photovoltaic strings, which is used to collect images of the faulty photovoltaic strings and send them to the three-dimensional digital twin model of the photovoltaic station after a photovoltaic string fault is identified;

[0051] Determine the faulty photovoltaic panel module, which is used to extract features from the collected image and match and compare it with the photovoltaic panel image feature library to determine the faulty photovoltaic panel;

[0052] The maintenance personnel positioning module is used to identify the maintenance personnel according to the image acquisition device to obtain the two-dimensional coordinates of the maintenance personnel in the image, convert the two-dimensional coordinates of the maintenance personnel into three-dimensional coordinates, and transmit them to the three-dimensional digital twin model;

[0053] The maintenance personnel path guidance module is used to utilize the three-dimensional digital twin model to conduct real-time interaction of the maintenance personnel's location information with the maintenance personnel's terminal equipment, plan the path for the maintenance personnel to reach the location of the faulty photovoltaic panel, and provide guidance.

[0054] As an improvement of the above system, the system further comprises:

[0055] Construct a three-dimensional digital twin model module to collect data from photovoltaic stations using image acquisition equipment; pre-process the collected images and then use three-dimensional modeling software to model them to generate a three-dimensional digital twin model of the photovoltaic station.

[0056] Compared with the prior art, the advantages of this application are:

[0057] This application uses the digital twin model as the core carrier, integrating the overall architecture of the functions of fault photovoltaic panel positioning and maintenance personnel positioning guidance. The digital twin model reflects the actual situation of the station in real time, closely links various links such as fault positioning, personnel position interaction and path planning, and realizes data sharing and collaborative work based on a unified platform. For example, when the station fault identification system identifies the faulty photovoltaic string, according to the fault string position information in the three-dimensional digital twin model, the image acquisition device is mobilized to obtain the white light and infrared images of the fault string, identify the faulty photovoltaic panel in the fault string, and immediately plan the optimal path based on the position of the maintenance personnel in the three-dimensional digital twin model after locating the faulty photovoltaic panel, and update all aspects of information in real time. Such a structure brings about the beneficial effects of smoother overall operation and maintenance process and tighter and more efficient cooperation in various links, ensuring the reliability and timeliness of the entire process from fault discovery to maintenance and processing of the photovoltaic station, and effectively improving the operation and maintenance management level of the station. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Shown is a flow chart of the photovoltaic station fault inspection and positioning method based on digital twins. DETAILED DESCRIPTION

[0059] The technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0060] Example 1

[0061] like Figure 1 As shown, the present application provides a photovoltaic station fault inspection and positioning method based on digital twins, including:

[0062] Step 1: Generate a 3D digital twin model

[0063] Step 1-1: Plan the drone scanning path and use the drone to collect data from the photovoltaic station:

[0064] Develop a drone scanning path plan based on the scale, terrain, and distribution of photovoltaic modules of the photovoltaic station. Divide multiple scanning areas according to the shape and area of ​​the station to ensure the coverage of the drone scanning. During the scanning process, the drone is equipped with sensor equipment such as laser radar and optical camera. The laser radar is used to measure the spatial data such as the three-dimensional coordinates and distance of the photovoltaic panel, and capture the outline and relative position information of the photovoltaic string and photovoltaic panel; the optical camera is used to obtain the appearance details.

[0065] Step 1-2: Process the data collected by the drone and build a 3D model of the photovoltaic station:

[0066] After the drone collects the raw data, it performs data preprocessing. The lidar data is filtered and denoised, and the outlier removal algorithm based on spatial density clustering is used to calculate the local spatial density of each point in the point cloud data. Points with a density significantly lower than the surrounding points are considered outliers. Based on the geometric characteristics and distribution patterns of the point cloud, the remaining points are screened for a second time to remove isolated points caused by local occlusion or reflection anomalies. For areas where the original data quality is too poor, a second scanning collection is performed.

[0067] The model is reconstructed using 3D modeling software combined with processed data. The geometric framework of the basic terrain, strings and photovoltaic panels of the photovoltaic station is constructed through point cloud data, and the image information is mapped to the corresponding geometric model to generate a 3D digital twin model of the photovoltaic station. In the digital twin model, each photovoltaic panel corresponds to a unique virtual entity, and the properties of the photovoltaic panel such as position, shape, and angle are accurately matched with the actual situation, providing a reliable digital basic platform for subsequent fault analysis and positioning.

[0068] Steps 1-3: Calibrate and optimize the 3D model of the photovoltaic station:

[0069] To ensure the accuracy of the digital twin model, after the construction is completed, multiple known coordinate reference points in the station (such as fixed markers at the station boundary, landmark positions of specific strings, box transformers, monitoring poles, etc.) are selected, and field measurements are carried out using high-precision measuring instruments such as total stations to compare the coordinates of corresponding points in the calibration model. For deviations, the spatial transformation algorithm is used to fine-tune and optimize the model as a whole, handle data uncertainty, and improve model accuracy.

[0070] According to the subsequent expansion and renovation of photovoltaic stations, the newly collected drone data is matched with the existing model for local features, and point cloud data (geometric features such as normal vectors and curvatures and shape descriptor features) and image data (grayscale co-occurrence matrix texture, local binary pattern and feature point features) are extracted respectively. First, an initial rough match is performed, and possible matching areas are screened according to location information through space division and index establishment; then, an accurate match is performed, and feature descriptor distance measurement and graph structure matching optimization are used to identify the changed areas (such as the newly added photovoltaic panel area, the equipment layout area after the renovation, etc.), and the model is rebuilt and fused only for the changed areas to reduce the resource consumption of model update.

[0071] Step 2: Collect the fault string image

[0072] Step 2-1: Select the image acquisition equipment and plan the location of the image acquisition equipment:

[0073] According to the environmental characteristics of the photovoltaic station (such as lighting conditions, temperature range, wind and sand conditions, etc.) and the accuracy requirements for fault detection, select white light cameras and infrared cameras with appropriate accuracy. If fixed cameras are used, they are arranged according to the string arrangement and distribution density, using the principle of combining uniform distribution with key area density. In addition, considering the wind direction and sunshine direction of the photovoltaic station, the camera is installed on the leeward side and the side that is not directly exposed to sunlight to reduce the erosion and interference of wind and sand and strong light on the camera lens.

[0074] The cameras carried by drones still need to meet the above performance requirements so that they can flexibly shoot strings in specific areas from different aerial perspectives.

[0075] Or a combination of fixed cameras and drones can be adopted, with drones supplementing possible blind spots of fixed cameras and improving image information of fault conditions that are difficult to collect with fixed-position cameras.

[0076] Step 2-2: Set and control the parameters of the image acquisition device:

[0077] For daylight cameras, set automatic exposure, white balance and other parameter adjustment strategies according to the changes in light intensity at different time periods during the day. When the light is dark in the morning and evening, appropriately increase the exposure time and sensitivity to ensure that the image is not too dark; during the strong light period at noon, adjust the aperture size and shutter speed to avoid overexposure and ensure that the details on the surface of the photovoltaic panel are clearly visible.

[0078] For the temperature measurement range and emissivity parameter settings of the infrared camera, a temperature measurement parameter library is established in combination with the material properties of the photovoltaic panels and the actual operating temperature historical data. The corresponding temperature measurement range and emissivity parameters are automatically matched according to the model information, ambient temperature and humidity, light intensity, wind speed and scheduling output of the monitored photovoltaic panels to improve the accuracy of temperature measurement.

[0079] After receiving the image acquisition instruction for the faulty string, the digital twin platform uniformly controls the fixed camera or drone camera to capture images.

[0080] Step 2-3: Collect, transmit and store faulty PV string images:

[0081] When the station fault identification system identifies a faulty photovoltaic string, the image acquisition device is mobilized to obtain white light and infrared images of the faulty string based on the location information of the faulty string in the three-dimensional digital twin model. The station fault identification system is an existing fault identification system that can use a variety of fault analysis methods to discover the fault conditions of photovoltaic strings, such as photovoltaic string power generation detection and judgment, neural network model data judgment, etc. After the station fault identification system detects that a photovoltaic string has failed, it can provide the number information of the failed photovoltaic string to the three-dimensional digital twin model. Photovoltaic strings are composed of several photovoltaic panels. The existing technology can generally only detect the faulty photovoltaic string, and cannot accurately locate which photovoltaic panel in the photovoltaic string has failed.

[0082] The image data collected by fixed cameras and drone cameras are transmitted to the digital twin platform in real time through wired and wireless communication methods. An image database is established on the platform to classify and store images according to information such as acquisition time, string number, camera location (different identifiers for distinguishing fixed cameras and drone cameras), which is convenient for subsequent retrieval and call. A data redundancy backup mechanism is set up, and distributed storage technology is used to back up image data to multiple storage nodes to prevent data loss due to single point failures, ensure the integrity and availability of image data, and provide stable data support for faulty photovoltaic panel positioning based on image recognition.

[0083] Step 3: Locate the faulty photovoltaic panel based on image recognition

[0084] Step 3-1: Preprocess the image and extract features:

[0085] The image of the faulty string is retrieved from the image database and preprocessed. The white light image is processed as follows: grayscale processing, which converts the color image into a grayscale image to simplify subsequent calculations and highlight the texture features of the image; histogram equalization, which enhances the image contrast; and denoising filtering, which improves image quality. The infrared image is processed as follows: temperature correction, which eliminates temperature measurement errors caused by factors such as ambient temperature changes and the thermal stability of the camera itself; thermal imaging enhancement, which enhances the contrast between the hot spot and the surrounding normal area.

[0086] Image feature extraction algorithms are used to extract multi-dimensional feature information from preprocessed images. Shape features of photovoltaic panels (such as rectangular outlines, corner features, etc., which are accurately obtained through edge detection algorithms such as the Canny edge detection algorithm), texture features (using gray-level co-occurrence matrix and other methods to calculate texture roughness, directionality and other parameters) and color features (using color histograms and other methods to count the distribution of different colors) are extracted from white light images; from infrared images, a multi-scale analysis method based on wavelet transform is used to obtain the macroscopic temperature gradient distribution of hot spots and the microscopic temperature fluctuation details inside the hot spots.

[0087] Step 3-2: Feature matching and fault judgment:

[0088] The extracted faulty string image features are matched and compared with the photovoltaic panel image feature library. In the matching process, similarity calculation algorithms (such as Euclidean distance, cosine similarity, etc.) are used to measure the degree of difference between image features. When a photovoltaic panel has obvious deformation in shape features (the similarity with normal shape features is lower than the set threshold), disordered texture features (texture parameters exceed the normal fluctuation range), abnormal color (significant difference from the normal color histogram), or hot spots appear in the infrared image (hot spot area, temperature gradient and other indicators do not meet the standards of normal fault-free state), the photovoltaic panel is judged to be a faulty photovoltaic panel, and the position of the faulty photovoltaic panel is calibrated in the three-dimensional digital twin model.

[0089] Step 4: Obtain the location information of the maintenance personnel

[0090] Step 4-1: Layout and selection of image acquisition equipment:

[0091] If it is a fixed camera, high-definition image acquisition equipment should be reasonably arranged at key locations within the station (such as channel entrances, intersections, main work areas, etc.) to avoid direct sunlight affecting the image quality. The camera can fully capture the full body features of the maintenance personnel and avoid obstructions that affect the image quality.

[0092] If the camera is mounted on a drone, the location information of the maintenance personnel can be obtained based on the time and location of the maintenance personnel entering the photovoltaic area.

[0093] Step 4-2: Extract and identify the maintenance personnel image features and determine the maintenance personnel's two-dimensional coordinates:

[0094] After the image acquisition device acquires the image inside the station in real time, the target detection algorithm is used to extract and identify the features of the maintenance personnel in the image. In addition to the maintenance personnel, the acquired image has at least 4 three-dimensional coordinate reference points with known three-dimensional coordinates. The reference points can be objects such as poles, stones, specific photovoltaic panels and box transformers; to ensure that the location information of the maintenance personnel can be three-dimensionally located. The image acquisition speed with a qualified number of three-dimensional coordinate reference points is the personnel position update frequency.

[0095] The selected target detection algorithm is trained using the maintenance personnel image data, so that the model learns the appearance characteristics of maintenance personnel under different postures, different clothing, different lighting conditions (such as human body contours, color distribution, clothing characteristics, etc.), and improves the accuracy and uniqueness of recognition by identifying specific logos, color stripes and other features on the maintenance personnel's work clothes, avoiding misjudgment with other objects or people. The collected images are input into the trained algorithm model, and the model quickly and accurately locates the maintenance personnel in the image.

[0096] After the maintenance personnel are identified, the two-dimensional relative position of the maintenance personnel and the coordinate reference point in the image is calculated. The lower left corner vertex of the image is selected as the origin, and the two boundaries of the vertex are used as the x-axis and y-axis in the rectangular coordinate system. The origin coordinates are marked as (x min ,y min ), the diagonal coordinate is (x max ,y max ).

[0097] The positions of personnel and calibration objects in the image are marked with bounding boxes, and the center point of the bounding box is used as the two-dimensional positioning information. The two-dimensional coordinates (u, v) of the maintenance personnel in the image can be obtained using the following formula:

[0098]

[0099] Where: x pmin 、x pmax ,y pmin ,y pmax They are the minimum x-axis value, maximum x-axis value, minimum y-axis value, and maximum y-axis value of the maintenance personnel boundary box, respectively. The two-dimensional coordinate calculation of the calibration object in the image is similar.

[0100] Step 4-3: Convert the maintenance personnel's two-dimensional coordinates into three-dimensional position coordinates:

[0101] In order to input the location information of the maintenance personnel into the three-dimensional digital twin model, the three-dimensional coordinates of the reference point and the two-dimensional relative coordinates of the reference point and the maintenance personnel in the image are used to realize the mapping of the two-dimensional image coordinates of the maintenance personnel to the three-dimensional space coordinates.

[0102] Assume that the coordinates of the point on the image plane are (u, v), the coordinates of the point in the three-dimensional space are (X, Y, Z), and the perspective transformation matrix is ​​M (3×4). The relationship can be expressed as:

[0103]

[0104] The perspective transformation matrix M can be expressed as:

[0105]

[0106] M contains the camera internal parameters (focal length f x 、f y and the principal point coordinates c x 、c y ) and external parameters (rotation matrix R and translation vector T), the expression is:

[0107]

[0108] In the above formula: r ij is the element of the rotation matrix R, t i are the elements of the translation vector T.

[0109] Solving the perspective transformation matrix M based on the three-dimensional coordinates of the reference point and the two-dimensional coordinates in the image requires four or more reference points, by establishing the following set of equations:

[0110]

[0111] Substitute the coordinates of 4 or more reference points into the above equations to obtain a linear equation system. Solve it to get the value of the perspective transformation matrix M. Substitute the two-dimensional relative coordinates of the maintenance personnel and the perspective transformation matrix M into the following formula to solve the three-dimensional coordinates of the maintenance personnel.

[0112]

[0113] There are 4 or more coordinate reference points in the image. Too many and unevenly distributed reference points will introduce redundancy and complexity, resulting in unstable results, so the reference points are screened.

[0114] Calculate the spatial distance between the reference point and other reference points, reference point P i and P j The spatial distance is:

[0115]

[0116] Construct the reference point distance matrix D of the current image, element D ij =d ij , calculate each reference point P iThe distance distribution entropy of other reference points is divided into several intervals according to the set rules (such as uniform division). ij The frequency of falling into the kth interval is recorded as f ik , calculate the reference point P by the following formula i The distance distribution entropy H i :

[0117]

[0118] Reference point P i The distance distribution entropy H i The larger the value, the more representative the point is for the overall spatial structure.

[0119] In addition, the distance between the reference point and the maintenance personnel is used as another basis for selecting the reference point, which helps to reduce the measurement error. The four reference points with the largest distance distribution entropy in the image are brought into the solution to obtain the initial value of the perspective transformation matrix M0, and the initial value of the maintenance personnel's three-dimensional position information (X p0 , Y p0 , Z p0 ), reference point P i The initial distance d from the maintenance personnel's three-dimensional position information ip0 for:

[0120]

[0121] Combine the reference point distance distribution entropy and the distance between the reference point and the initial value of the three-dimensional position of the person to calculate the selection tendency S of each reference point i0 :

[0122]

[0123] Among them, α is the weight coefficient, which can be adjusted according to the actual situation to balance the importance of distribution entropy and personnel distance; the larger the distribution entropy and the closer the personnel distance, the higher the selection tendency of the point.

[0124] The four reference points with the highest selection tendency are selected to recalculate the perspective transformation matrix M1, and the three-dimensional position information of the maintenance personnel is calculated through the perspective transformation matrix M1.

[0125] Iterating the above process multiple times can improve the accuracy of the three-dimensional position of the personnel, and the number of iterations is selected according to the actual situation.

[0126] Step 5: Realize real-time interaction between the 3D digital twin model and the maintenance personnel’s terminal device to guide the maintenance personnel to find the faulty PV module

[0127] Step 5-1: Real-time update and interaction mechanism of location information:

[0128] The three-dimensional location information of the maintenance personnel is transmitted to the three-dimensional digital twin model of the photovoltaic station in real time, and their current location is dynamically displayed in the model through specific identification (such as using icons of different colors to represent different maintenance personnel). At the same time, the digital twin platform establishes a real-time communication connection with the terminal devices (such as tablets, smart phones, etc.) held by the maintenance personnel, and synchronously pushes the location information in the model to the terminal devices, so that the maintenance personnel can intuitively see their position in the entire station model and the location of other relevant personnel (if there are multiple people performing maintenance work at the same time). The operations of the maintenance personnel on the terminal equipment (such as confirming that they have arrived at a certain location, feedback on the actual situation on site, etc.) are also fed back to the digital twin platform in real time, updating the corresponding status information in the model, realizing two-way real-time location information interaction, and ensuring that the information held by all parties is always consistent and updated in a timely manner.

[0129] The three-dimensional location information of maintenance personnel is encrypted, stored and transmitted in the blockchain network. Each node (including the digital twin platform, maintenance personnel terminal equipment, etc.) ensures the consistency and non-tamperability of the information through a consensus mechanism.

[0130] Step 5-2: Path planning and guidance display:

[0131] In the digital twin system, based on the current location of the maintenance personnel and the location of the faulty photovoltaic panel, combined with the actual layout of existing roads, passages, buildings, equipment, etc. within the photovoltaic station, a path planning algorithm is used to consider distance factors, road conditions (such as whether there are obstacles, whether construction is in progress, etc.) and safety factors (such as avoiding high-voltage areas, dangerous equipment, etc.) to calculate the shortest and most convenient route from the current location of the maintenance personnel to the faulty photovoltaic panel.

[0132] The planned path information is presented to the maintenance personnel in a visual way (e.g., the path is marked with eye-catching lines on the map interface of the maintenance personnel's terminal device, and key information such as the direction and distance are informed through voice prompts, text prompts, etc.). The volume, speed and tone of the voice prompts, as well as the color and size of the text prompts are adjusted according to the fatigue of the maintenance personnel to improve the acceptability of the guidance information and guide the maintenance personnel to quickly and accurately reach the location of the faulty photovoltaic panel through the optimal route for fault repair.

[0133] Through real-time and accurate location interaction and guidance mechanism, maintenance personnel do not need to spend a lot of time looking for fault points, which effectively improves the overall efficiency of fault repair and reduces the time waste and labor cost increase caused by searching for fault locations.

[0134] The photovoltaic station fault inspection and positioning method based on digital twins provided in this application is based on a three-dimensional digital twin model. After determining that a string has a fault, the fault string image is collected through a reasonably arranged white light camera and an infrared camera (including a fixed camera and a dispatchable drone-mounted camera), and image preprocessing, feature extraction and matching algorithms are used to comprehensively analyze multi-dimensional image features to accurately locate the faulty photovoltaic panel in the faulty string. This method, which combines drone scanning and image recognition technology, breaks through the limitations of traditional fault detection methods in the precision level positioning of a single photovoltaic panel, greatly improves the accuracy and efficiency of fault positioning, can quickly lock the faulty photovoltaic panel, and effectively assists in the subsequent maintenance work. This method does not rely on satellite positioning technologies such as GPS and Beidou. Based on digital twins, it uses image recognition technology to incorporate maintenance personnel into the three-dimensional digital twin model of the photovoltaic station to achieve accurate positioning of maintenance personnel. Through image acquisition equipment (including fixed cameras and on-demand drone cameras), the image features of maintenance personnel are extracted with the help of deep learning target detection algorithms, and the position of maintenance personnel in the image is accurately converted to the three-dimensional digital twin model by establishing a mapping relationship between two-dimensional image coordinates and three-dimensional space coordinates, so as to achieve accurate three-dimensional positioning within the station. Based on the real-time interactive maintenance personnel location information of the digital twin model, the optimal travel path is planned in combination with the actual layout of the station to guide maintenance personnel to the location of the faulty photovoltaic panel, which effectively solves the problem of accurate positioning and rapid guidance of maintenance personnel in the complex environment of photovoltaic stations, reduces the time consumption of manually finding fault points in faulty photovoltaic strings, and improves the overall fault repair efficiency.

[0135] Example 2

[0136] The present application also provides a photovoltaic station fault inspection and positioning system based on digital twins, which is implemented based on the above method, and the system includes:

[0137] The module for collecting images of faulty photovoltaic strings is used to collect images of the faulty photovoltaic strings and send them to the three-dimensional digital twin model of the photovoltaic station after a fault is identified in the photovoltaic strings.

[0138] The faulty photovoltaic panel determination module is used to extract features from the collected images and match and compare them with the photovoltaic panel image feature library to determine the faulty photovoltaic panel.

[0139] The maintenance personnel positioning module is used to identify the maintenance personnel according to the image acquisition equipment to obtain the two-dimensional coordinates of the maintenance personnel in the image, convert the two-dimensional coordinates of the maintenance personnel into three-dimensional coordinates, and transmit them to the three-dimensional digital twin model.

[0140] The maintenance personnel path guidance module is used to utilize the three-dimensional digital twin model to conduct real-time interaction of the maintenance personnel's location information with the maintenance personnel's terminal equipment, plan the path for the maintenance personnel to reach the location of the faulty photovoltaic panel, and provide guidance.

[0141] Construct a three-dimensional digital twin model module to collect data from photovoltaic stations using image acquisition equipment; pre-process the collected images and then use three-dimensional modeling software to model them to generate a three-dimensional digital twin model of the photovoltaic station.

[0142] The present application may also provide a computer device, comprising: at least one processor, a memory, at least one network interface and a user interface. The various components in the device are coupled together through a bus system. It is understood that the bus system is used to achieve connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.

[0143] The user interface may include a display, a keyboard or a pointing device, such as a mouse, a trackball, a touch pad or a touch screen.

[0144] It is understood that the memory in the embodiments disclosed in the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0145] In some embodiments, the memory stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and applications.

[0146] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present disclosure can be included in the application.

[0147] In the above embodiment, the processor may also call a program or instruction stored in the memory, specifically, a program or instruction stored in an application program, and is used to:

[0148] Execute the steps of the above method.

[0149] The above method can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The above-disclosed methods, steps and logic block diagrams can be implemented or executed. The general processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the above-disclosed method can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0150] It is understood that the embodiments described in the present application can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0151] For software implementation, the technology of the present application can be implemented by executing the functional modules (such as procedures, functions, etc.) of the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0152] The present application may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, each step in the above method embodiment can be implemented.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. A photovoltaic station fault inspection and positioning method based on digital twins, comprising: When a photovoltaic string failure is identified, the image acquisition device collects images of the faulty photovoltaic string and sends them to the three-dimensional digital twin model of the photovoltaic station; Extract features from the collected images and compare them with the photovoltaic panel image feature library to identify the faulty photovoltaic panel; The image acquisition device identifies the maintenance personnel and obtains the two-dimensional coordinates of the maintenance personnel in the image, converts the two-dimensional coordinates of the maintenance personnel into three-dimensional coordinates, and transmits them to the three-dimensional digital twin model; The three-dimensional digital twin model interacts with the maintenance personnel’s terminal devices in real time to monitor the maintenance personnel’s location information, plan a path for the maintenance personnel to reach the location of the faulty photovoltaic panel, and provide guidance.

2. The photovoltaic station fault inspection and positioning method based on digital twin according to claim 1 is characterized in that: Also includes: Use image acquisition equipment to collect data from photovoltaic stations; After preprocessing the collected images, three-dimensional modeling software is used to generate a three-dimensional digital twin model of the photovoltaic station.

3. The photovoltaic station fault inspection and positioning method based on digital twin according to claim 1 is characterized in that: The image acquisition device acquires images of the faulty photovoltaic string, including white light images and infrared images.

4. The photovoltaic station fault inspection and positioning method based on digital twin according to claim 3 is characterized in that: The step of determining a failed photovoltaic panel comprises: Performing grayscale processing, histogram equalization processing and denoising filtering processing on the white light image; extracting shape features, texture features and color features of the photovoltaic panel from the white light image as image features; Performing temperature correction and thermal imaging enhancement processing on the infrared image; using a multi-scale analysis method based on wavelet transform on the infrared image to obtain the macroscopic temperature gradient distribution of the hot spot and the microscopic temperature fluctuation inside the hot spot as image features; The extracted image features are matched and compared with the photovoltaic panel image feature library, and the similarity calculation method is used to determine the faulty photovoltaic panel.

5. The photovoltaic station fault inspection and positioning method based on digital twin according to claim 1 is characterized in that: The image acquisition device identifies the maintenance personnel and obtains the two-dimensional coordinates of the maintenance personnel in the image, including: Using the trained object detection model, the maintenance personnel in the image are identified to obtain the bounding box of the maintenance personnel in the image; Calculate the two-dimensional coordinates (u, v) of the maintenance personnel in the image according to the coordinates of the bounding box: Among them, x pmin 、x pmax ,y pmin ,y pmax They are the minimum x-axis value, maximum x-axis value, minimum y-axis value, and maximum y-axis value of the maintenance personnel boundary box respectively.

6. The photovoltaic station fault inspection and positioning method based on digital twin according to claim 5 is characterized in that: The step of converting the two-dimensional coordinates of the maintenance personnel into three-dimensional coordinates comprises: Create a 3×4 dimensional perspective transformation matrix M: Select n, n ≥ 4 reference points, and substitute the coordinates of the reference points into the following formula to calculate the element values ​​of the perspective transformation matrix M of the two-dimensional coordinate solution: Among them, (u i , v i ) represents the two-dimensional coordinates of the i-th reference point; (X i , Y i , Z i ) represents the three-dimensional coordinates of the i-th reference point; The three-dimensional coordinates of the maintenance personnel are calculated by the following formula: Among them, (X, Y, Z) represents the three-dimensional coordinates of the maintenance personnel.

7. The photovoltaic station fault inspection and positioning method based on digital twin according to claim 6 is characterized in that: The reference point screening method comprises: Calculate the spatial distance between each reference point and other reference points: Among them, d ij Represents the spatial distance between the i-th reference point and the j-th reference point; Calculate the distance distribution entropy between each reference point and other reference points: Among them, H i represents the distance distribution entropy of the i-th reference point; the spatial distance is evenly divided into several intervals, f ik Indicates the distance d ij The frequency of falling into the kth interval; Obtain n reference points with the largest distance distribution entropy as the reference points for final screening.

8. The photovoltaic station fault inspection and positioning method based on digital twin according to claim 7 is characterized in that: The step of converting the two-dimensional coordinates of the maintenance personnel into three-dimensional coordinates also includes: The initial value of the perspective transformation matrix M0 is calculated based on the reference point n with the largest distance distribution entropy in the image, and the initial value of the three-dimensional position of the maintenance personnel (X p0 , Y p0 , Z p0 ); Calculate the distance between other reference points and the initial value of the maintenance personnel's three-dimensional position: Among them, d ip0 Indicates the distance between the i-th reference point and the initial value of the maintenance personnel's three-dimensional position; Calculate the selection preference for each reference point: Among them, S i0 represents the selection tendency of the i-th reference point; α represents the weight coefficient; Select n reference points with the highest selection tendency and recalculate the perspective transformation matrix M1, and calculate the three-dimensional coordinates of the maintenance personnel through the perspective transformation matrix M1; Repeat the above process a set number of times to improve the accuracy of the maintenance personnel's three-dimensional coordinates.

9. A photovoltaic station fault inspection and positioning system based on digital twins, implemented based on any method described in claims 1-8, characterized in that: The system comprises: A module for collecting images of faulty photovoltaic strings, which is used to collect images of the faulty photovoltaic strings and send them to the three-dimensional digital twin model of the photovoltaic station after a photovoltaic string fault is identified; Determine the faulty photovoltaic panel module, which is used to extract features from the collected image and match and compare it with the photovoltaic panel image feature library to determine the faulty photovoltaic panel; A maintenance personnel positioning module, used to identify the maintenance personnel according to the image acquisition device to obtain the two-dimensional coordinates of the maintenance personnel in the image, convert the two-dimensional coordinates of the maintenance personnel into three-dimensional coordinates, and transmit them to the three-dimensional digital twin model; and The maintenance personnel path guidance module is used to utilize the three-dimensional digital twin model to conduct real-time interaction of the maintenance personnel's location information with the maintenance personnel's terminal equipment, plan the path for the maintenance personnel to reach the location of the faulty photovoltaic panel, and provide guidance.

10. The photovoltaic station fault inspection and positioning system based on digital twin according to claim 9 is characterized in that: The system further comprises: Construct a three-dimensional digital twin model module to collect data from photovoltaic stations using image acquisition equipment; pre-process the collected images and then use three-dimensional modeling software to model them to generate a three-dimensional digital twin model of the photovoltaic station.

Images