Control method and device for paving equipment, storage medium and paving system

Abstract

The invention provides a control method and device for paving equipment, a storage medium and a paving system.The paving equipment comprises a mechanical arm, a camera and a bearing part, the bearing part is used for bearing at least one photovoltaic panel, and the method comprises the steps that the camera is controlled to collect a first image set, and a second image set is obtained based on the first image set; determining a safe position set of the mechanical arm and a reference position in the safe position set, controlling the mechanical arm to grab the photovoltaic panel in the bearing part, controlling the mechanical arm to move the photovoltaic panel to the reference position based on the safe position set, and controlling the mechanical arm to move the photovoltaic panel from the reference position to a target position of the first paving frame, the efficiency and the accuracy of photovoltaic panel paving can be improved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a control method, device, storage medium and paving system for paving equipment. Background Art

[0002] At present, solar photovoltaic panels are a kind of power generation device that generates current when exposed to sunlight. Photovoltaic panels can be made into different shapes, and the components can be connected to each other to generate more electricity. In the construction of photovoltaic power stations, how to lay photovoltaic panels has become a hot topic of research. In the existing technology, photovoltaic panels are usually carried and installed manually, and the efficiency and accuracy of photovoltaic panel laying are low. Summary of the invention

[0003] The embodiments of the present application provide a control method, device, storage medium and paving system for paving equipment, which realizes the paving of photovoltaic panels through images collected by cameras, thereby improving the efficiency and accuracy of photovoltaic panel paving and improving the safety of photovoltaic panel movement.

[0004] In the first aspect, the embodiment of the present application provides a control method for a paving device, wherein the paving device includes a mechanical arm, a camera, and a carrier, wherein the carrier is used to carry at least one photovoltaic panel, and the method includes: controlling the camera to collect a first image set; based on the first image set, determining the safe position set of the mechanical arm and the reference position in the safe position set; controlling the mechanical arm to grab the photovoltaic panel in the carrier; based on the safe position set, controlling the mechanical arm to move the photovoltaic panel to the reference position; and controlling the mechanical arm to move the photovoltaic panel from the reference position to the target position of the first paving frame. In this way, the safe position set of the mechanical arm and the reference position in the safe position set are identified through image analysis, and the mechanical arm is controlled to move the photovoltaic panel from the reference position to the target position, thereby improving the efficiency and accuracy of photovoltaic panel paving.

[0005] In a possible implementation, the determination of the safe position set of the robot arm and the reference position in the safe position set based on the first image set includes: identifying each image in the first image set to obtain the second image set of the first paving frame and the third image set of other objects; determining the free position information of the first paving frame based on the second image set, and determining the safe position set of the robot arm based on the third image set; determining the target position from the area corresponding to the free position information; and determining the reference position from the safe position set based on the target position. In this way, the target position is determined from the area corresponding to the free position information through image analysis and recognition, thereby further determining the reference position, thereby improving the efficiency and accuracy of determining the reference position.

[0006] In a possible implementation, determining the safe position set of the robotic arm based on the third image set includes: determining the object position of the other object based on the third image set; determining the mobile position set of the robotic arm based on the attribute information of the robotic arm; excluding the object position from the mobile position set to obtain the safe position set of the robotic arm. In this way, the safe position set of the robotic arm is obtained by excluding the object position from the mobile position set based on the third image set, thereby improving the safety of the robotic arm movement.

[0007] In a possible implementation, the determining of the reference position from the safety position set based on the target position includes: determining a reference safety position corresponding to the target position from the safety position set, the number of the reference safety positions being greater than or equal to 1; obtaining an offset value between the reference safety position and the midpoint of the target position, the offset value including a horizontal offset value and a vertical offset value; and selecting a reference safety position from the reference safety positions whose horizontal offset value is less than a first threshold value and whose vertical offset value is less than a second threshold value as a reference position in the safety position set. In this way, by selecting a reference safety position whose horizontal offset value and vertical offset value both meet preset conditions from the reference safety positions as a reference position in the safety position set, the efficiency of determining the reference position is improved.

[0008] In a possible implementation, if there is no reference safety position whose horizontal offset value is less than the first threshold value and whose vertical offset value is less than the second threshold value, a reference safety position whose horizontal offset value is less than the first threshold value or whose vertical offset value is less than the second threshold value is selected from the reference safety positions as the first reference position; the movement of the manipulator is controlled according to the horizontal offset value of the first reference position, so that the vertical offset value between the photovoltaic panel grasped by the manipulator after the manipulator stops moving and the midpoint of the target position is less than the first threshold value. In this way, the movement of the manipulator is controlled according to the horizontal offset value of the first reference position, so that the vertical offset value between the photovoltaic panel grasped by the manipulator after the manipulator stops moving and the midpoint of the target position is less than the first threshold value, thereby improving the accuracy of the movement of the manipulator.

[0009] In a possible implementation, if it is determined based on the second image set that the first paving frame has no free position, a second paving frame to be paved is determined; and the paving equipment is controlled to move from the first position of the first paving frame to the second position of the second paving frame. In this way, if the first paving frame has no free position, the paving equipment is controlled to move from the first position to the second position, thereby avoiding wasting the time of the paving equipment and improving the paving efficiency of the paving equipment.

[0010] In a possible implementation, the control of the paving equipment to move from the first position of the first paving frame to the second position of the second paving frame includes: obtaining the first position of the first paving frame and the second position of the second paving frame; controlling the camera to collect a fourth image set; determining a moving path between the first position and the second position based on the fourth image set; and controlling the paving equipment to move from the first position of the first paving frame to the second position of the second paving frame according to the moving path. In this way, the moving path is identified through image analysis, and the paving equipment is controlled to move from the first position to the second position, thereby improving the efficiency of the movement of the paving equipment.

[0011] In a second aspect, an embodiment of the present application provides a control device for paving equipment, comprising a unit for indicating a method provided in the first aspect or any embodiment of the first aspect.

[0012] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or computer instructions are stored. When the computer program or the computer instructions are executed by a processor, the method described in the first aspect or any one of the first aspects is implemented.

[0013] In a fourth aspect, an embodiment of the present application provides a paving system, comprising a paving frame, a photovoltaic panel and the paving equipment described in the first aspect or any one of the first aspects, wherein the paving equipment is used to move the photovoltaic panel to a target area of ​​the photovoltaic panel in the paving frame.

[0014] Implementing the embodiments of the present application has the following beneficial effects:

[0015] The safe position set of the robotic arm and the reference position in the safe position set are determined based on the first image set captured by the camera. After the robotic arm grabs the photovoltaic panel in the carrier, the robotic arm can be controlled to move the photovoltaic panel to the reference position based on the safe position set, and then the robotic arm can be controlled to move the photovoltaic panel from the reference position to the target position of the photovoltaic panel in the paving frame. In this way, the photovoltaic panel is installed through the images captured by the camera, which can improve the efficiency and accuracy of the photovoltaic panel installation. And moving the photovoltaic panel based on the safe position set of the robotic arm can improve the safety of the movement of the photovoltaic panel. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1A schematic diagram of an application scenario of a control method for paving equipment provided in an embodiment of the present application;

[0018] Figure 2 A schematic diagram of a paving frame carrying photovoltaic panels provided in an embodiment of the present application;

[0019] Figure 3 A flow chart of a control method for paving equipment provided in an embodiment of the present application;

[0020] Figure 4 A schematic structural diagram of a control device for paving equipment provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The terms "first" and "second" in the specification, claims and drawings of this application are only used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0022] "At least one (item)" means one or more, "more than one" means two or more, "at least two (items)" means two or three and more than three, and "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c".

[0023] The embodiment of the present application provides a paving system, which includes a paving device, a paving frame and a photovoltaic panel. In the embodiment of the present application, the paving device is used to move the photovoltaic panel to the target area of ​​the photovoltaic panel in the paving frame. Specifically, it can be applied to application scenarios such as photovoltaic panel cleaning scenarios and photovoltaic panel replacement scenarios that require the position of the photovoltaic panel to be moved, and the application scenarios are not limited here.

[0024] This application does not limit the devices in the paving system. In addition to paving equipment, paving frames and photovoltaic panels, the paving system may also include other devices such as auxiliary vehicles. Among them, the auxiliary vehicle can be used to carry people or other mechanical arms to assist in placing the photovoltaic panels on the target position of the paving frame.

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] Please refer to Figure 1 , Figure 1 The present invention provides a schematic diagram of an application scenario of a control method for paving equipment, which includes a construction worker 101, paving equipment 102, a photovoltaic panel 103 and a server 104. Figure 1 As shown, the paving equipment can be used as follows Figure 1 The form of the vehicle is shown, and the power system of the vehicle can be an electric system, a fuel system, a hybrid system, etc. This application does not limit the power type of the vehicle, and does not limit the driving method of the paving equipment.

[0027] It should be noted that Figure 1 The number and shape of each device in the application scenario diagram shown, as well as the number of construction workers 101 are used for example and do not constitute a limitation on the embodiments of the present application. One construction worker 101 can operate multiple paving devices 102, and one paving device 102 can pave multiple photovoltaic panels 103.

[0028] For the driving method of paving equipment, the following are given Figure 1 Two forms not shown, of which:

[0029] Form 1: A driver for operating the power system is set inside the paving equipment, and the paving equipment is driven by a handheld driver or an unmanned driving strategy.

[0030] Form 2: A remote controller adapted to the paving equipment is provided so as to control the movement of the paving equipment through the remote controller of the paving equipment.

[0031] Optionally, the remote controller may be a physical remote controller, on which buttons related to the operation type of the paving equipment are arranged. For example, movement keys for controlling the paving equipment to turn left, turn right, move forward, move backward, etc. Another example is buttons for controlling the operation of devices such as a robotic arm and a carrier, etc. This application does not limit the button types of the remote controller.

[0032] Optionally, the remote controller may be a virtual remote controller of the electronic device.

[0033] Among them, the electronic devices can be smart phones, tablet computers, laptops, PDAs, mobile Internet devices (MID), wearable devices (such as smart watches, smart bracelets, etc.), smart voice interaction devices, smart home appliances (such as smart TVs, etc.) and vehicle-mounted devices, etc., without limitation here.

[0034] The electronic device may be a server, which may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, without limitation.

[0035] The virtual remote controller may be provided with virtual buttons for the above-mentioned related functions, or may be provided with trigger gestures for related functions, etc., and this application does not limit the implementation of the virtual remote controller. It can be understood that by using the remote controller to trigger the paving equipment to operate, the control efficiency can be improved.

[0036] It should be noted that the above two forms are only examples. In fact, there may be other forms of devices to drive the paving equipment to move.

[0037] Please refer to Figure 2 , Figure 2 A schematic diagram of a paving frame carrying photovoltaic panels provided in an embodiment of the present application. Figure 2 As shown, the figure includes a paving frame 20 , a first-layer paving frame 201 , a second-layer paving frame 202 , a support 203 and a photovoltaic panel 103 , and the photovoltaic panel 103 can be placed on the paving frame 20 .

[0038] In the embodiment of the present application, the paving equipment is used to place the photovoltaic panel 103 on the paving frame 20. The paving equipment can also be used to move the photovoltaic panel 103 in the paving frame 20 to other locations, which is not limited here. The paving frame 20 is used to place at least one photovoltaic panel 103, and the placement area corresponding to the paving frame 20 includes the target area of ​​each photovoltaic panel 103.

[0039] The present application does not limit the location of the target area of ​​the photovoltaic panel 103, which can be determined according to a preset placement rule. Among them, the preset placement rule can be used to determine the placement order and placement position of the photovoltaic panel 103 placed on the paving frame 20. The placement rule can be to place the photovoltaic panel 103 from left to right or from right to left; or in the case where the paving frame 20 includes a multi-layer sub-paving frame, the photovoltaic panels 103 can be placed layer by layer from left to right and then the photovoltaic panels 103 of the next layer, or the photovoltaic panels 103 can be placed layer by layer from right to left and then the photovoltaic panels 103 of the next layer, or one photovoltaic panel 103 can be placed on each layer and then one photovoltaic panel 103 can be placed on the next layer, etc.

[0040] In the embodiment of the present application, at least two placement rules are included. For example, the photovoltaic panels 103 of the upper paving frame 20 are placed from left to right, and the photovoltaic panels 103 of the lower paving frame 20 are placed from right to left, so that the upper paving frame 20 can be laid first, and then the lower paving frame 20, or the lower paving frame 20 can be laid first, and then the upper paving frame 20, which is conducive to improving the paving efficiency.

[0041] In the embodiment of the present application, the paving frame 20 can be as follows Figure 2 The structure shown includes at least a first-layer paving frame 201 and a second-layer paving frame 202 , so that at least two layers of photovoltaic panels 103 can be placed.

[0042] In the embodiment of the present application, the paving frame 20 includes a support member 203 to keep the photovoltaic panel 103 at a certain distance from the ground. The present application does not limit the form of the support member 203, which can be as follows: Figure 2 Shown in upright position, or can be Figure 2 The form of the tripod not shown can also be in the form of wheels, etc. The paving frame 20 in the form of wheels is conducive to movement. The paving frame 20 in the form of a tripod has high stability.

[0043] exist Figure 2 In the embodiment, a paving frame 20 is used as an example. In fact, the paving equipment can place different photovoltaic panels 103 on different paving frames 20. The paving equipment and the paving frame 20 can also adopt other forms.

[0044] For the sake of simplicity, they are not described one by one in the accompanying drawings. The structure of the paving equipment is described in detail below. The paving equipment may include a camera, a robotic arm, and a carrier. For example, the number of cameras may be one or more, the number of robotic arms may be one or more, and the number of carriers may be one or more.

[0045] The present application does not limit the location of the processor, which can be located at any position of the paving equipment, or set in the above-mentioned electronic equipment. The processor is used to control the paving equipment or the devices or components in the paving equipment to work.

[0046] In the embodiment of the present application, the carrier is used to carry at least one photovoltaic panel 103, and specifically, at least one photovoltaic panel 103 can be carried in the gripping area of ​​the carrier. The present application does not limit the number of photovoltaic panels 103 carried in the carrier.

[0047] In the embodiment of the present application, the carrier may also include Figure 1 The lifting device not shown is used to lift the photovoltaic panel 103 of the carrier to the gripping area of ​​the carrier, so that the robot arm can move the photovoltaic panel 103 from the gripping area to other areas, such as the target area of ​​the photovoltaic panel 103 in the laying frame.

[0048] The lifting device may be a combination of a motor and a base plate, similar to the function of an elevator. Photovoltaic panels 103 are stacked on the base plate, and the motor is used to lift the photovoltaic panels 103 on the base plate to a preset height. The preset height may be the thickness of the photovoltaic panel 103, and the area corresponding to the preset height is called the grabbing area. In this way, each time a photovoltaic panel 103 is grabbed, the motor lifts the base plate by the thickness of a photovoltaic panel 103, so that the next photovoltaic panel 103 can be grabbed in the grabbing area.

[0049] In the embodiment of the present application, the robot arm can be as follows Figure 1 As shown, it is arranged in the middle position of the paving equipment, and the bearing member can be Figure 1 As shown, it is set at both ends of the paving equipment. Figure 1 In a form not shown in the figure, for example, the mechanical arm can be arranged at both ends of the paving equipment, and the bearing member can be arranged at the middle position of the paving equipment. For another example, the mechanical arm and the bearing member are separately arranged on the body of the paving equipment, or the mechanical arm and the bearing member are connected and arranged on the body of the paving equipment.

[0050] The present application does not limit the number of mechanical arms and carriers, for example, the number of mechanical arms is 1, and the number of carriers is 2. In this way, while the photovoltaic panel 103 of one carrier is used for mounting, the photovoltaic panel 103 in another carrier is used for unmounting, which can improve work efficiency.

[0051] In the embodiment of the present application, the paving equipment may also include Figure 1 The support member not shown is used to support the robot arm, which is convenient for the robot arm to move, so as to avoid the robot arm from hitting other devices and being damaged when moving. Other devices may include a protective shell including a power system.

[0052] In an embodiment of the present application, the camera may be located on a robotic arm, or may be located elsewhere. In some feasible examples, the camera may be located in the environment of the paving system (paving equipment and / or photovoltaic panel 103). For example, a real-time panoramic camera, etc. In an embodiment of the present application, the camera is used to capture images, and specific reference may be made to the methods described below, which will not be described in detail here. The embodiment of the present application does not limit the type, position, and number of cameras, and the number may be at least one.

[0053] In the embodiment of the present application, the robotic arm is used to move the position, specifically, it can be moved to the grasping position of the carrier, the reference position of the laying frame 20, the target position of the photovoltaic panel 103, etc., and the method described later can be referred to and will not be described in detail here.

[0054] In an embodiment of the present application, the mechanical arm may include a mechanical upper arm, a mechanical lower arm, a first rotation axis, and a second rotation axis. The first rotation axis connects the mechanical upper arm and the paving equipment, and is used to rotate the mechanical upper arm so that the mechanical upper arm rotates relative to the paving equipment. The second rotation axis connects the mechanical upper arm and the mechanical lower arm, and is used to rotate the mechanical lower arm so that the mechanical lower arm rotates relative to the mechanical upper arm. The path formed by rotating the mechanical upper arm by the first rotation axis and / or rotating the mechanical lower arm by the second rotation axis can be called a moving path.

[0055] In an embodiment of the present application, the mechanical arm may further include a clamp connected to the mechanical upper arm. The clamp is used to grab the photovoltaic panel 103, and may also be used to place the photovoltaic panel 103. For example, the clamp grabs the photovoltaic panel 103 from the grabbing area of ​​the carrier, and maintains the grabbing state until the photovoltaic panel 103 reaches its target area. The clamp may include a plurality of suction cups, and the photovoltaic panel 103 is grasped by the suction force of the suction cups. Alternatively, the clamp may include a plurality of magnets, and the photovoltaic panel 103 is attracted by the magnetic force of the magnets, etc. The present application does not limit the type of the clamp.

[0056] In an embodiment of the present application, the mechanical arm may further include a moving frame connected to the mechanical upper arm and the clamping member. The moving frame is used to move the clamping member to extend the position of the mechanical upper arm so that the clamping member can be moved to a farther position. The moving frame may be realized by a servo motor.

[0057] In the embodiment of the present application, the mechanical arm may further include a turntable connected to the clamping member. The turntable may be used to rotate the angle of the clamping member, for example, 180 degrees, so that the clamping member can grasp the photovoltaic panel 103 from another direction without moving the mechanical upper arm and the mechanical lower arm.

[0058] The above structure is only an example. The robotic arm may also include other devices, and the paving equipment may also include a processor for controlling components in the paving equipment (such as a camera and a robotic arm, etc.) to implement corresponding methods.

[0059] In the embodiment of the present application, the processor may include but is not limited to a control board or a control chip. Exemplarily, the processor may include a battery management system (BMS), a micro control unit (MCU), a central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a pulse width modulator (PWM) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.

[0060] The working mode of the paving equipment will be described below in combination with the method.

[0061] Please refer to Figure 3 , Figure 3 The present invention provides a flow chart of a control method for a paving device provided in an embodiment of the present invention. The control method can be executed by a paving device or a processor in the paving device, and the paving device includes a mechanical arm, a camera, and a carrier. Figure 3 As shown, the method includes steps S301 to S305, wherein:

[0062] S301, controlling a camera to collect a first image set.

[0063] In an embodiment of the present application, the first image set includes environmental images captured when the position of the camera changes, or may include environmental images captured when the position of the camera remains unchanged, etc. When the camera is located on a robotic arm, the first image set can be captured by moving the robotic arm to improve the efficiency of capturing the first image set. The first image set includes one or more images. The present application does not limit the number of images in the first image set, which may be 2, 3, 100 or 200, etc. The present application does not limit the format of the images in the first image set. The resolution of the images in the first image set may be 640×400, 1024×600, 1920×1080, 3840×2160, etc.

[0064] In the embodiment of the present application, the camera can collect the first image set in real time, can take pictures during the real-time video recording process, or can collect the first image set at a preset time period (such as every second or every minute), or can collect images when a preset condition is met to obtain the first image set. Among them, the preset condition can be before paving the photovoltaic panel, such as receiving a paving instruction, etc. The present application does not limit the timing of the camera collecting the first image set.

[0065] S302: Determine a safe position set of the robot arm and a reference position in the safe position set based on the first image set.

[0066] In the embodiment of the present application, the reference position may belong to a safe position set and be located above the paving frame. The safe position set may include multiple safe positions, and the reference position is a safe position where the photovoltaic panel is located above the target position of the paving frame.

[0067] The safe position set includes the safe positions of the mobile manipulator and the photovoltaic panels grasped by the manipulator when the positions of objects in the environment of the paving equipment remain unchanged, for example, the positions that will not collide with other objects or paving frames, the positions that will not collide with other photovoltaic panels, etc. Optionally, the safe position set can also be pre-set according to the outsourced dimensions of the paving equipment. The outsourced dimensions of the paving equipment include the length of the paving equipment, the width of the paving equipment, and the height of the paving equipment. The outsourced dimensions of the paving equipment, the position parameters of the paving frames at the construction site, the position parameters of other objects at the construction site, and the position parameters of the photovoltaic panels are input into the paving equipment in advance. The paving equipment determines multiple safe positions of the mobile manipulator and the photovoltaic panels grasped by the manipulator based on the above information, and generates a safe position set to prevent the paving equipment from colliding with other objects, paving frames, and photovoltaic panels.

[0068] When the camera on the robot arm moves with the robot arm, the camera will also move with it. The moving position of the robot arm can be determined based on the coordinate system of the robot arm. The origin of the coordinate system can be the position of the robot arm in the initial state, that is, the position where the robot arm returns to the normal state. The moving position of the robot arm or the position of the camera in the robot arm can be determined by the position parameters corresponding to different coordinate axes in the coordinate system, such as the position parameters corresponding to the x-axis, y-axis and z-axis respectively. Among them, the x-axis, y-axis and z-axis can refer to Figure 2 The x-axis represents the width direction of the rectangle corresponding to the laying frame 20, the y-axis represents the length direction of the rectangle corresponding to the laying frame 20, and the z-axis represents the height direction of the laying frame 20. The moving position of the robot arm or the position of the camera in the robot arm can be represented by a matrix, which can be called a point pose matrix based on the robot arm base, which is used to represent the position relative to the origin of the coordinate system.

[0069] If the camera is not located on the robotic arm, the camera can be installed on the shell of the paving equipment close to the robotic arm. The reference coordinate system of the camera can still be the coordinate system of the robotic arm or other coordinate systems, and the origin of the coordinate system can be the position of the mounting base of the robotic arm.

[0070] In an embodiment of the present application, based on the first image set, a safe position set of the robot arm and a reference position in the safe position set are determined, including: identifying each image in the first image set to obtain a second image set of the first laying frame and a third image set of other objects; determining the free position information of the first laying frame based on the second image set, and determining the safe position set of the robot arm based on the third image set; determining the target position from the area corresponding to the free position information; and determining the reference position from the safe position set based on the target position.

[0071] In an embodiment of the present application, the position of an object in an image of the first image set may be equal to the product of the first matrix and the second matrix. The first matrix may represent the position of the camera relative to the origin of the coordinate system when the camera captures the first image. The second matrix may represent the position of the object in the image relative to the position of the camera. The product of the first matrix and the second matrix may be used to represent the position of the object in the image relative to the origin of the coordinate system.

[0072] In an embodiment of the present application, a first recognition model can be established, and the first recognition model is used to recognize objects and attributes of objects in an image, and the attributes of objects include the shape of the object and the color of the object. The first recognition model is used to recognize each image in the first image set, and a second image set of the first paving frame and a third image set of other objects are obtained. For example, the first recognition model is used to recognize the paving frame and other objects, and a training set is established. The training set includes multiple different images of paving frames and other objects. The first recognition model is trained using the training set to obtain a second recognition model, and the second recognition model is used to recognize the first image set, thereby improving the precision and accuracy of recognition.

[0073] In an embodiment of the present application, the second image set is a collection of images of the first paving frame, and the second image set includes one or more images. Based on the information of the first paving frame in the second image set, the vacant position information of the first paving frame in the second image set is identified, and the vacant position information includes the position information of the vacant installation positions and the number information of the vacant installation positions, and no photovoltaic panels are installed on the vacant installation positions.

[0074] In an embodiment of the present application, the second recognition model is also used to identify the position of the first paving frame and the attributes of the first paving frame in the second image set, and the attributes of the first paving frame include the shape of the first paving frame and the color of the first paving frame. The position of the first paving frame in the second image set is determined using the second recognition model. The present application does not limit the number of images in the second image set, which may be 2, 3, 100 or 200, etc. The present application does not limit the format of the images in the second image set. The resolution of the images in the second image set may be 640×400, 1024×600, 1920×1080, 3840×2160, etc.

[0075] In an embodiment of the present application, the third image set is a collection of images of other objects, and the third image set includes one or more images. The second recognition model is also used to identify the positions of other objects in the third image set and the attributes of other objects, and the attributes of other objects include the shapes of other objects and the colors of other objects. The present application does not limit the number of images in the third image set, which may be 2, 3, 100 or 200, etc. The present application does not limit the format of the images in the third image set. The resolution of the images in the third image set may be 640×400, 1024×600, 1920×1080, 3840×2160, etc.

[0076] According to the positions and volumes of other objects in the third image set, a safe position set of the robot arm is determined, and the safe position set includes safe position information and volume information of the safe position. In the embodiment of the present application, there are multiple free installation positions in the area corresponding to the free position information, and the reference position where the robot arm will not collide with other objects when moving can be determined from the multiple free installation positions in order of distance from the robot arm from near to far, combined with information such as the positions and volumes of other objects.

[0077] In an embodiment of the present application, based on the first image set, determining the safe position set of the robot arm and the reference position in the safe position set may also include: identifying each image in the first image set to obtain a second image set of the paving frame and an environmental image set of the paving environment; determining the free position information of the paving frame based on the second image set, and determining the dangerous position of the environment in which the paving frame is located based on the environmental image set; determining the target position from the area corresponding to the free position information, and determining the reference position above the target position; and determining the safe position set of the robot arm based on the dangerous position.

[0078] Among them, the second image set of the paving frame includes the image of the paving frame. The environmental image set of the paving environment may include the image of the paving frame, or may not include the image of the paving frame. The environmental image set of the paving environment may include images of photovoltaic panels, images of paving equipment and other images of objects in the environment where the paving equipment is located. Among them, the images of objects in the environment where the paving equipment is located may include images of dangerous objects. The dangerous object may be an object that is easily collidable, etc. The present application can identify objects in the first image set, which will include paving frames, robotic arms, photovoltaic panels and other objects.

[0079] In an embodiment of the present application, the placement order of each free position is determined, and the placement order can be from left to right in the horizontal direction, or from right to left; and / or from near to far; the free position that is placed first in the placement order is determined as the target position.

[0080] The camera parameters of the camera, such as focal length, pixel, sensitivity, exposure time, frame rate, etc., will affect the shooting range of the captured image. The position parameters of the camera, such as tilt, etc., will affect the shooting range of the captured image. In this way, the position of the object in the image relative to the camera when the camera captures an image in the first image set can be determined based on the camera parameters and position parameters of the camera. Then, based on the position of the camera relative to the origin of the coordinate system and the position of the object in the image relative to the camera, the position of the object in the image can be determined.

[0081] Each image in the first image set is recognized to obtain the second image set of the first laying frame and the third image set of other objects, then the free position information of the first laying frame is determined, and then the safe position set of the robot arm is determined, and further, the target position is determined from the area corresponding to the free position information, and finally, the reference position is determined from the safe position set based on the target position, and the target position is determined from the area corresponding to the free position information through image analysis and recognition, so as to further determine the reference position, thereby improving the efficiency and accuracy of determining the reference position.

[0082] Determining a safe position set of the robotic arm based on the third image set includes: determining the object positions of other objects based on the third image set; determining a mobile position set of the robotic arm based on attribute information of the robotic arm; excluding the object positions from the mobile position set to obtain a safe position set of the robotic arm.

[0083] In an embodiment of the present application, the second recognition model is used to determine the object positions of other objects in the third image set. The attribute information of the robotic arm includes the length of the robotic arm, the width of the robotic arm, the height of the robotic arm, the volume of the robotic arm, and the moving speed of the robotic arm. The mobile position set is a collection of positions where the robotic arm moves, which reflects the moving path of the robotic arm. Excluding the object positions of other objects from the mobile position set can prevent the robotic arm from colliding with other objects during movement. The object positions of other objects are determined based on the third image set, and then the mobile position set of the robotic arm is determined based on the attribute information of the robotic arm. Then, the object position is excluded from the mobile position set to obtain a safe position set of the robotic arm. The object position is excluded from the mobile position set based on the third image set to obtain a safe position set of the robotic arm, thereby improving the safety of the robotic arm movement.

[0084] In an embodiment of the present application, determining a reference position from a safety position set based on a target position includes: determining a reference safety position corresponding to the target position from the safety position set, the number of reference safety positions being greater than or equal to 1; obtaining an offset value between the reference safety position and a midpoint of the target position, the offset value including a horizontal offset value and a vertical offset value; and selecting a reference safety position from the reference safety positions whose horizontal offset value is less than a first threshold and whose vertical offset value is less than a second threshold as a reference position in the safety position set.

[0085] In the embodiment of the present application, the reference safety position is a safety position corresponding to the target position, and the reference safety position may be 2, 3, 4, 5, etc. For example, the horizontal offset value is the distance between the midpoints of the safety position and the target position in the horizontal direction, and the vertical offset value is the distance between the midpoints of the safety position and the target position in the vertical direction.

[0086] A reference safety position corresponding to the target position is determined from the safety position set, the number of the reference safety positions being greater than or equal to 1, and then an offset value between the reference safety position and the midpoint of the target position is obtained, the offset value including a horizontal offset value and a vertical offset value, and then a reference safety position whose horizontal offset value is less than a first threshold and whose vertical offset value is less than a second threshold is selected from the reference safety positions as the reference position in the safety position set. By selecting a reference safety position whose horizontal offset value and vertical offset value both meet preset conditions from the reference safety positions as the reference position in the safety position set, the efficiency of determining the reference position is improved.

[0087] In an embodiment of the present application, a reference position is determined from a set of safe positions based on a target position, and further includes: if there is no reference safe position whose horizontal offset value is less than a first threshold and whose vertical offset value is less than a second threshold, a reference safe position whose horizontal offset value is less than the first threshold or whose vertical offset value is less than the second threshold is selected from the reference safe positions as the first reference position; and the movement of the robotic arm is controlled according to the horizontal offset value of the first reference position so that the vertical offset value between the photovoltaic panel grasped by the robotic arm after the robotic arm stops moving and the midpoint of the target position is less than the first threshold.

[0088] In the embodiment of the present application, the first threshold is determined according to the requirements of the construction, and the second threshold is determined according to the requirements of the construction. If there is no reference safety position whose horizontal offset value is less than the first threshold and whose vertical offset value is less than the second threshold, a reference safety position whose horizontal offset value is less than the first threshold is selected from the reference safety positions as the first reference position, and the size relationship between the vertical offset value and the second threshold is not limited. If there is no reference safety position whose horizontal offset value is less than the first threshold and whose vertical offset value is less than the second threshold, a reference safety position whose vertical offset value is less than the second threshold is selected from the reference safety positions as the first reference position, and the size relationship between the horizontal offset value and the first threshold is not limited.

[0089] If there is no reference safety position with a horizontal offset value less than the first threshold and a vertical offset value less than the second threshold, a reference safety position with a horizontal offset value less than the first threshold or a vertical offset value less than the second threshold is selected from the reference safety positions as the first reference position. Then, the movement of the robotic arm is controlled according to the horizontal offset value of the first reference position so that the vertical offset value between the photovoltaic panel grasped by the robotic arm after the robotic arm stops moving and the midpoint of the target position is less than the first threshold. The movement of the robotic arm is controlled according to the horizontal offset value of the first reference position so that the vertical offset value between the photovoltaic panel grasped by the robotic arm after the robotic arm stops moving and the midpoint of the target position is less than the first threshold, thereby improving the accuracy of the movement of the robotic arm.

[0090] In an embodiment of the present application, based on the first image set, a safe position set of the robot arm and a reference position in the safe position set are determined, and it also includes: if it is determined based on the second image set that there is no vacant position on the first laying frame, a second laying frame to be laid is determined; and the laying equipment is controlled to move from the first position of the first laying frame to the second position of the second laying frame.

[0091] In the embodiment of the present application, the first position is the position of the first paving frame, the second position is the position of the second paving frame, and the second paving frame is the paving frame to be paved. In the embodiment of the present application, the second recognition model is used to identify the second image set to determine that there is no free position on the first paving frame, then the photovoltaic panel cannot be installed on the first paving frame, and then the second recognition model is used to identify the second paving frame to be paved in the second image set, and there is a free position on the second paving frame.

[0092] In the embodiment of the present application, when controlling the paving equipment to move from the first position to the second position, the control method can be wired control or wireless control. For example, the speed of the paving equipment moving from the first position to the second position can be adjusted according to demand. When the terrain is undulating, the speed can be slower, and when the terrain is flat, the speed can be faster.

[0093] If it is determined based on the second image set that there is no vacant position on the first paving frame, determine the second paving frame to be paved, and then control the paving equipment to move from the first position of the first paving frame to the second position of the second paving frame. If there is no vacant position on the first paving frame, control the paving equipment to move from the first position to the second position, thereby avoiding wasting time of the paving equipment and improving the paving efficiency of the paving equipment.

[0094] In an embodiment of the present application, controlling the paving equipment to move from a first position of a first paving frame to a second position of a second paving frame includes: obtaining the first position of the first paving frame and the second position of the second paving frame; controlling the camera to capture a fourth image set; based on the fourth image set, determining a moving path between the first position and the second position; and controlling the paving equipment to move from the first position of the first paving frame to the second position of the second paving frame according to the moving path.

[0095] In the embodiment of the present application, the first position of the first paving frame and the second position of the second paving frame are obtained by using a camera, and the first position of the first paving frame and the second position of the second paving frame can be photographed in real time, or the first position and the second position can be photographed at preset intervals. Alternatively, the first position and the second position can be photographed when an instruction is received within a preset period of time.

[0096] In an embodiment of the present application, the fourth image set includes multiple images from the image at the first position to the second position, and the moving path is determined from the multiple images in the fourth image set. When determining the moving path, the moving path will not cause the paving equipment to collide with the first paving frame or the second paving frame or other objects.

[0097] In an embodiment of the present application, a moving path with the shortest distance and no collision can be selected from multiple moving paths. In an embodiment of the present application, the present application does not limit the number of images in the fourth image set, which can be 2, 3, 100 or 200, etc. For example, the present application does not limit the format of the images in the fourth image set. For example, the resolution of the images in the fourth image set can be 640×400, 1024×600, 1920×1080, 3840×2160, etc.

[0098] The first position of the first paving frame and the second position of the second paving frame are obtained, and then the camera is controlled to collect a fourth image set. Further, based on the fourth image set, a moving path between the first position and the second position is determined. Finally, according to the moving path, the paving equipment is controlled to move from the first position to the second position. The moving path is identified through image analysis, and the paving equipment is controlled to move from the first position to the second position, thereby improving the efficiency of the movement of the paving equipment.

[0099] S303, controlling the robotic arm to grab the photovoltaic panel in the carrier.

[0100] In an embodiment of the present application, the robot arm includes a body and at least one suction cup, at least one suction cup is connected to the body, the suction cup grasps the photovoltaic panel through the suction force of the suction cup and firmly fixes the photovoltaic panel, the body drives the suction cup to move, the suction cup drives the photovoltaic panel to move, so that the photovoltaic panel moves in the horizontal direction or the vertical direction to be taken out from the carrier. For example, the robot arm may also include a clamping member connected to the shell of the paving equipment, the clamping member may include a plurality of magnets, the photovoltaic panel is attracted by the magnetic force of the magnets, and the photovoltaic panel is stably fixed, the magnet drives the photovoltaic panel to move, so that the photovoltaic panel moves in the horizontal direction or the vertical direction to be taken out from the carrier, and the present application does not limit the type of the clamping member.

[0101] The robot arm is controlled to grab the photovoltaic panels in the carrier, thereby improving the efficiency of grabbing the photovoltaic panels.

[0102] S304: Based on the safety position set, control the robotic arm to move the photovoltaic panel to a reference position.

[0103] In an embodiment of the present application, the paving equipment can be moved in the form of a vehicle, and the robotic arm is rotatably connected to the vehicle. The robotic arm can rotate and move on the vehicle along a first direction, a second direction or a third direction to move the mobile photovoltaic panel to a reference position. The first direction, the second direction and the third direction are all perpendicular to each other, and the vehicle can also drive the robotic arm to move along the first direction or the second direction.

[0104] In the embodiment of the present application, the speed of controlling the robotic arm to move the photovoltaic panel can be adjusted according to demand. When the terrain is undulating, the speed can be slower, and when the terrain is flat, the speed can be faster.

[0105] Based on the safe position set, the robotic arm is controlled to move the photovoltaic panel to the reference position, which improves the safety of moving the photovoltaic panel.

[0106] S305: Control the robotic arm to move the photovoltaic panel from the reference position to the target position of the first laying frame.

[0107] In an embodiment of the present application, the control body is rotated or extended to drive the suction cup to move, and the suction cup drives the photovoltaic panel to move, so as to move the photovoltaic panel from the reference position to the target position of the first laying frame, and then the suction cup breaks through the air to place the photovoltaic panel at the target position.

[0108] Understandably, Figure 3 In the method shown, the safe position set of the robotic arm and the reference position in the safe position set are identified through image analysis, and the robotic arm is controlled to move the photovoltaic panel from the reference position to the target position, thereby improving the efficiency and accuracy of photovoltaic panel installation.

[0109] See also Figure 4 , Figure 4 300 is a schematic diagram of a control device for paving equipment provided in an embodiment of the present application. Based on the above system architecture, the control device 300 for paving equipment can be a server or a module in a server. The control device 300 for paving equipment includes at least: a collection module 301 and a processing module 302, wherein:

[0110] The acquisition module 301 is used to control the camera to acquire a first image set.

[0111] The processing module 302 is used to determine the safe position set of the robotic arm and the reference position in the safe position set based on the first image set; control the robotic arm to grab the photovoltaic panel in the carrier; based on the safe position set, control the robotic arm to move the photovoltaic panel to the reference position; control the robotic arm to move the photovoltaic panel from the reference position to the target position of the first laying frame.

[0112] In one possible example, the processing module 302 is used to identify each image in the first image set to obtain a second image set of the first laying frame and a third image set of other objects; determine the free position information of the first laying frame based on the second image set, and determine the safe position set of the robotic arm based on the third image set; determine the target position from the area corresponding to the free position information; and determine the reference position from the safe position set based on the target position.

[0113] In one possible example, the processing module 302 is used to determine the object positions of other objects based on the third image set; determine the mobile position set of the robotic arm based on the attribute information of the robotic arm; and exclude the object positions from the mobile position set to obtain a safe position set of the robotic arm.

[0114] In one possible example, the processing module 302 is used to determine a reference safety position corresponding to the target position from the safety position set, where the number of reference safety positions is greater than or equal to 1; obtain an offset value between the reference safety position and a midpoint of the target position, where the offset value includes a horizontal offset value and a vertical offset value; and select a reference safety position from the reference safety positions whose horizontal offset value is less than a first threshold and whose vertical offset value is less than a second threshold as a reference position in the safety position set.

[0115] In one possible example, the processing module 302 is used to select a reference safety position with a horizontal offset value less than the first threshold value or a vertical offset value less than the second threshold value from the reference safety positions as the first reference position if there is no reference safety position with a horizontal offset value less than the first threshold value and a vertical offset value less than the second threshold value; and control the movement of the robotic arm according to the horizontal offset value of the first reference position so that the vertical offset value between the photovoltaic panel grasped by the robotic arm and the midpoint of the target position after the robotic arm stops moving is less than the first threshold value.

[0116] In a possible example, the processing module 302 is used to determine a second paving frame to be paved if it is determined based on the second image set that there is no free position on the first paving frame; and control the paving equipment to move from the first position of the first paving frame to the second position of the second paving frame.

[0117] In one possible example, the processing module 302 is used to obtain the first position of the first paving frame and the second position of the second paving frame; control the camera to collect a fourth image set; determine a moving path between the first position and the second position based on the fourth image set; and control the paving equipment to move from the first position of the first paving frame to the second position of the second paving frame according to the moving path.

[0118] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program or computer instructions are stored. When the computer program or computer instructions are executed by a processor, part or all of the steps of any method described in the method embodiment are implemented.

[0119] The present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute to implement some or all steps of any method described in the method embodiment. The computer program product can be a software installation package, and the computer includes an electronic device and a server.

[0120] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0121] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0122] In several embodiments of the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0123] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0124] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software program module.

[0125] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, RAM, ROM, mobile hard disks, magnetic disks or optical disks.

[0126] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A control method for paving equipment, characterized in that: The paving equipment includes a mechanical arm, a camera and a bearing member, the bearing member is used to carry at least one photovoltaic panel, and the method includes: Controlling the camera to collect a first image set; Determining a safe position set of the robotic arm and a reference position in the safe position set based on the first image set; Controlling the robotic arm to grab the photovoltaic panel in the carrier; Based on the safety position set, controlling the mechanical arm to move the photovoltaic panel to the reference position; The robotic arm is controlled to move the photovoltaic panel from the reference position to a target position of the first paving frame.

2. The control method according to claim 1, characterized in that: The step of determining the safe position set of the robotic arm and the reference position in the safe position set based on the first image set includes: Recognize each image in the first image set to obtain a second image set of the first paving frame and a third image set of other objects; Determine the free position information of the first laying frame based on the second image set, and determine the safe position set of the robot arm based on the third image set; Determine the target location within the area corresponding to the vacant location information; A reference position is determined from the set of safe positions based on the target position.

3. The control method according to claim 2, characterized in that: Determining the safe position set of the robotic arm based on the third image set includes: determining an object position of the other object based on the third image set; Determining a moving position set of the robotic arm based on the attribute information of the robotic arm; The object position is excluded from the mobile position set to obtain a safe position set of the robot arm.

4. The control method according to claim 2, characterized in that: The determining a reference position from the safety position set based on the target position comprises: Determine a reference safety position corresponding to the target position from the safety position set, wherein the number of the reference safety positions is greater than or equal to 1; Acquire an offset value between the reference safety position and a midpoint of the target position, wherein the offset value includes a horizontal offset value and a vertical offset value; A reference safety position whose horizontal offset value is smaller than a first threshold and whose vertical offset value is smaller than a second threshold is selected from the reference safety positions as a reference position in the safety position set.

5. The control method according to claim 4, characterized in that: Also includes: If there is no reference safety position whose horizontal offset value is smaller than the first threshold and whose vertical offset value is smaller than the second threshold, select one of the reference safety positions whose horizontal offset value is smaller than the first threshold or whose vertical offset value is smaller than the second threshold as the first reference position; The movement of the robotic arm is controlled according to the horizontal offset value of the first reference position, so that the vertical offset value between the photovoltaic panel grasped by the robotic arm and the midpoint of the target position after the robotic arm stops moving is less than the first threshold value.

6. The control method according to claim 2, characterized in that: Also includes: If it is determined based on the second image set that there is no free space on the first paving frame, determining a second paving frame to be paved; The paving equipment is controlled to move from a first position of the first paving frame to a second position of the second paving frame.

7. The control method according to claim 6, characterized in that: The controlling the paving equipment to move from the first position of the first paving frame to the second position of the second paving frame comprises: Acquire a first position of the first laying frame and a second position of the second laying frame; Controlling the camera to collect a fourth image set; determining a movement path between the first position and the second position based on the fourth image set; According to the moving path, the paving equipment is controlled to move from a first position of the first paving frame to a second position of the second paving frame.

8. A control device for paving equipment, characterized in that: Comprising means for indicating a method as claimed in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or a computer instruction. When the computer program or the computer instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A paving system, characterized in that: It comprises a laying frame, a photovoltaic panel and a laying device according to any one of claims 1 to 7, wherein the laying device is used to move the photovoltaic panel to a target area of ​​the photovoltaic panel in the laying frame.

Images