Mobile visual inspection station-viewpoint joint clustering and sequence planning method
By combining clustering and sequence planning methods, the layout and sequence of viewpoints and stations are optimized, which solves the problem of poor planning results in the detection of large and complex components and improves detection efficiency and planning efficiency.
Patent Information
- Application Number
- CN202310842141.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-11
- Publication Date
- 2025-12-12
- Estimated Expiration
- 2043-07-11
AI Technical Summary
In existing technologies for mobile visual inspection of large and complex components, the decoupling of the joint sequence planning of station and viewpoint leads to poor planning results and fails to effectively combine the inherent relationship between viewpoint layout and station layout, thus affecting inspection efficiency.
A joint clustering and sequence planning method for mobile visual detection stations and viewpoints is adopted. Through data encoding and decoding, population initialization, evolutionary strategy update, individual selection, crossover and mutation operations, combined with fitness calculation, the joint planning of viewpoints and stations is optimized to reduce movement costs.
It improves the efficiency of visual inspection, enhances the quality of planning results, and significantly reduces the running time of the planning process.
Smart Images
Figure CN116883509B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of mobile visual inspection planning, and particularly relates to a mobile visual inspection station-point joint clustering and sequence planning method. BACKGROUND
[0002] At present, for the appearance quality inspection of large and complex components in the fields of aerospace and shipbuilding, an effective scheme is based on a mobile visual robot technology integrating a mobile robot, a mechanical arm and a visual sensor, which has the advantages of wide-area movement, accurate arrival and efficient acquisition.
[0003] In order to improve the detection efficiency, planning is usually needed before detection. The planning includes three key parts: viewpoint layout planning, station layout planning and joint sequence planning. In order to collect visual data of the target area faster and better, a number of shooting poses need to be set for the sensor in advance, i.e. viewpoint layout planning. Since the three-dimensional model of the product to be inspected is known, the viewpoint layout planning is usually regarded as a set cover problem. The purpose is to find the best camera position that minimizes the perception cost under the coverage constraint. The cost here is mainly the number of viewpoints required. In the case of fewer viewpoints, the image capture time and image processing time will be reduced. Similarly, on the basis of viewpoint layout planning, in order to ensure that the planned viewpoints are reachable and the number of stations is as small as possible, the layout of the stations needs to be planned. On the basis of viewpoint layout and station layout planning, the sequence needs to be planned. At present, the work mainly studies these two sequence planning respectively, without considering the internal relationship. For example, in the viewpoint sequence planning, most researches are aimed at fixed mechanical arm bases. At this time, the viewpoint sequence planning is usually regarded as a traveling salesman problem, and the goal is to reduce the cost of robot movement between different viewpoints as much as possible under the premise of no collision.
[0004] At present, in order to solve the station-point joint sequence planning problem, a typical scheme is to divide it into two planning steps: clustering planning, i.e. determining the station to which each viewpoint belongs; sequence planning, i.e. on the basis of the known station-point attribution relationship, modeling the sorting problem as a clustering traveling salesman problem for solving. Since there is no sequence information in the clustering planning stage, the optimization goal of this stage can only be approximate motion cost minimization, which deviates from the actual motion cost and may lead to poor quality of the final planning result. SUMMARY
[0005] The present application aims to: in the process of station-point joint clustering and sequence planning for mobile visual inspection, in order to overcome the problem of poor quality of the planning result caused by decoupling and approximation in the existing scheme, and further improve the efficiency of mobile visual inspection, the present application provides a station-point joint clustering and sequence planning method for mobile visual inspection.
[0006] The application is mainly realized by the following technical scheme: a mobile visual detection station-point joint clustering and sequence planning method, characterized by comprising the following steps:
[0007] Data encoding and decoding. The length of the chromosome is equal to the number of viewpoints participating in the sorting. For each gene, the encoding rule is as follows: the integer part represents the current viewpoint sequence number, and the decimal part represents the station sequence number to which the current viewpoint belongs. For the viewpoints belonging to the same station, their sequences follow the order of appearance in the chromosome. The station sequence follows the order of the first appearance position. Therefore, the head segment of the chromosome is used to represent the station sequence.
[0008] Population initialization. Set the population size; according to the encoding rule, randomly initialize each individual in the population. Set the number of iterations, and the following steps are iteratively performed.
[0009] Evolution strategy update. Here, three Boolean variables are used to control the subsequent crossover and mutation operations. The three Boolean variables correspond to clustering, station sequence and viewpoint sequence. For example, if the first variable is TRUE, clustering crossover and clustering mutation are performed; otherwise, they are not performed. According to the number of iterations, the entire evolution process is divided into three stages. In the first stage, the following strategies are alternately performed: (1) only optimize the clustering; (2) only optimize the station sequence. In the second stage, the following strategies are alternately performed: (1) only optimize the clustering; (2) only optimize the viewpoint sequence. In the third stage, the following strategies are alternately performed: (1) only optimize the clustering; (2) optimize the station sequence and the viewpoint sequence simultaneously; (3) optimize the clustering, the station sequence and the viewpoint sequence simultaneously.
[0010] Individual selection. According to the principle that the greater the fitness, the greater the selection probability, two individuals are selected from the population for subsequent crossover operation. The selected individuals are the parents, and the individuals after crossover are the offspring.
[0011] Individual crossover. It contains three independent sub-operations, which are respectively aimed at clustering, station sequence and viewpoint sequence. In one evolution process, these sub-operations are not all executed, but are controlled by the updated evolution strategy.
[0012] Clustering crossover, which is aimed at clustering planning. Clustering crossover exchanges the decimal part of the corresponding gene to update the belonging station of the current viewpoint. Specifically, when the genes at a certain position are exchanged, it should be judged first whether the viewpoint and the station after the exchange are reachable. If not, the genes at the position do not participate in the exchange.
[0013] Position sequence crossover, i.e., crossover facing position sequence. Decode the position sequence first, treat it as the crossover operation of classic traveling salesman problem, and re-encode the crossover result to get the chromosome after crossover.
[0014] Viewpoint sequence crossover, i.e., crossover facing viewpoint sequence. The chromosome fragments participating in viewpoint sequence crossover need to come from the same position; at this time, the crossover operation of viewpoint sequence also needs the process of decoding and re-encoding. In the two crossover chromosome fragments, if the viewpoint set belonging to the same position is inconsistent, the intersection needs to be taken first, and then the above-mentioned way is crossed. According to this rule, all positions are traversed, and the corresponding viewpoint sequence is crossed.
[0015] Individual mutation. Similar to the crossover operation, the mutation operation is still decoupled into cluster-oriented, position sequence-oriented, and viewpoint sequence-oriented sub-operations. These sub-operations are not all executed in one evolution process, but are controlled by the updated evolution strategy.
[0016] Cluster mutation. A gene is randomly selected from the chromosome; the viewpoint sequence number is decoded from the decimal part of the gene; a different position from the current one is randomly selected from the position set corresponding to the viewpoint, and it is encoded to update the decimal part of the gene.
[0017] Position sequence mutation. Decode the position sequence first, treat it as the mutation operation of classic traveling salesman problem, i.e., randomly select two positions and exchange their positions to realize mutation. And re-encode the mutation result to get the mutated chromosome.
[0018] Viewpoint sequence mutation. Decode the viewpoint sequence of a position, and randomly select two viewpoints in the decoded viewpoint sequence to exchange their positions. Then, re-encode the mutated sequence. According to this rule, all positions are traversed, and the corresponding viewpoint sequence is crossed.
[0019] Population update. Decode all chromosomes to get their cluster information, position sequence information, and viewpoint sequence information. Based on this information, fitness calculation is performed. The fitness of all individuals in the population is calculated, then the individuals are sorted in descending order of fitness, and the current optimal individual is obtained. If the current optimal individual is better than the global optimal individual, update the global optimal individual. The fitness is the inverse of the motion cost. The calculation method of the motion cost is as follows:
[0020] Let the position sequence be The number of positions is The viewpoint sequence belonging to the i-th position is The number of viewpoints is where the j-th viewpoint is denoted as
[0021] The total inter-viewpoint motion cost is the sum of four terms: the first term is the intra-station viewpoint motion cost, the second term is the inter-station viewpoint motion cost, the third term is the cost from the HOME viewpoint to the start viewpoint, and the last term is the cost from the final viewpoint to the HOME viewpoint. v (·) is used to calculate the motion cost between two viewpoints. Its formula is:
[0022]
[0023] The inter-station motion cost is:
[0024]
[0025] where s H is the HOME station.c s (·) is used to calculate the motion cost between two stations.
[0026] The present application has the following advantages: The present application is directed to mobile visual inspection, and can integrally realize clustering and sequence planning, improve the quality of the solution scheme, and further improve the visual inspection efficiency. Meanwhile, the proposed evolution strategy also significantly reduces the running time of the planning program. The present application can be applied to large and complex component mobile visual inspection scenes and other scenes with similar characteristics. BRIEF DESCRIPTION OF DRAWINGS
[0027] Figure 1 is the mobile visual inspection station-viewpoint joint clustering and sequence planning flowchart of the present application.
[0028] Figure 2 is the gene and chromosome coding schematic diagram of the present application.
[0029] Figure 3 is the update evolution strategy flowchart of the present application.
[0030] Figure 4 is the chromosome clustering crossover schematic diagram of the present application.
[0031] Figure 5 is the chromosome station sequence crossover schematic diagram of the present application.
[0032] Figure 6 is the chromosome viewpoint sequence crossover schematic diagram of the present application. DETAILED DESCRIPTION
[0033] The present application will be further described in detail below in combination with the drawings and examples.
[0034] The test sample used has dimensions of 3338*1804*718mm, with 22 stations and 301 viewpoints. The implementation of this invention includes the following steps ( Figure 1 ):
[0035] Data encoding and decoding. For example... Figure 2 As shown, the length of the chromosome is equal to the number of viewpoints involved in the sequencing. For each gene, the encoding rules are as follows: the integer part represents the current viewpoint number, and the fractional part represents the station number to which the current viewpoint belongs. Generally, the fractional part multiplied by 1000 gives the station index number. For viewpoints belonging to the same station, their sequences follow the order of their appearance on the chromosome. The station sequences, in turn, follow the order of their first appearance. Therefore, the head segment of the chromosome is used to represent the station sequence.
[0036] Initialization. Set the population size; randomly initialize each individual in the population according to the encoding rules. During initialization, known information includes: all viewpoint index numbers and the set of station indexes corresponding to each viewpoint. The iteration count is set to 6000, and the following steps are performed iteratively.
[0037] Evolutionary strategy update. For example... Figure 3 Here, three Boolean variables (S1, S2, S3) control the subsequent crossover and mutation operations. These three Boolean variables correspond to the clustering, station sequence, and viewpoint sequence, respectively. For example, if S1 is TRUE, then clustering crossover and clustering mutation are performed subsequently; otherwise, they are not performed. The entire evolutionary process is divided into three stages according to the number of iterations. In the first stage, the following strategies are performed alternately: (1) optimize only clustering; (2) optimize only station sequence. In the second stage, the following strategies are performed alternately: (1) optimize only clustering; (2) optimize only viewpoint sequence. In the third stage, the following strategies are performed alternately: (1) optimize only clustering; (2) optimize both station sequence and viewpoint sequence simultaneously; (3) optimize clustering, station sequence, and viewpoint sequence simultaneously. Specifically, Figure 3 The parameters can be set as follows: N1 = 2000, N2 = 4000, M = 30, R1 = 15, R2 = 25.
[0038] Individual selection. Following the principle that higher fitness corresponds to a higher selection probability, two individuals are selected from the population for subsequent crossover. The selected individuals become the parents, and the crossover result becomes the offspring. Specifically, individual selection can be based on the roulette wheel algorithm.
[0039] Individual crossover. The crossover operation is performed according to the aforementioned evolutionary strategy. It includes three independent sub-operations, targeting clustering, position sequence, and viewpoint sequence respectively.
[0040] Clustering crossover. For example...Figure 4 As shown in FIG. 6, the cluster crossover exchanges the fractional part of the genes to update the belonging station of the current viewpoint. Specifically, when the gene pair at a certain position is exchanged, it should be first determined whether the viewpoint and the station after the exchange are reachable. If not, the gene at the position does not participate in the exchange.
[0041] Station sequence crossover. As shown in FIG. 7, the station sequence is first decoded from the chromosome segment, and then regarded as the crossover operation of the classic traveling salesman problem, and the crossover result is re-encoded to obtain the chromosome after the crossover. Figure 5
[0042] Viewpoint sequence crossover. As shown in FIG. 8, for a certain station, all viewpoints belonging to the station are found from the two individuals to be crossed, and decoded into a viewpoint sequence. The crossover operation of the viewpoint sequence also needs the decoding and re-encoding process, and its principle is the same as the station sequence crossover. It should be noted that the viewpoint sets corresponding to the two individuals are not necessarily completely the same. If the viewpoint sets belonging to the same station are inconsistent, the intersection should be taken first, and then crossed in the above manner. According to this rule, all stations are traversed, and the corresponding viewpoint sequences are crossed. Figure 6
[0043] Individual mutation. Similar to the crossover operation, the mutation operation also needs to follow the evolutionary strategy, and is also decoupled into cluster-oriented, station sequence-oriented, and viewpoint sequence-oriented mutation sub-operations.
[0044] Cluster mutation. A gene is randomly selected from the chromosome; the viewpoint number is decoded from the fractional part of the gene; a station different from the current one is randomly selected from the station set corresponding to the viewpoint, and is encoded to update the fractional part of the gene.
[0045] Station sequence mutation. The station sequence is first decoded, and regarded as the mutation operation of the classic traveling salesman problem, that is, two stations are randomly selected and their positions are exchanged to realize mutation. And the mutation result is re-encoded to obtain the chromosome after mutation.
[0046] Viewpoint sequence mutation. The chromosome segment participating in the viewpoint sequence mutation comes from the same station. For a certain station, the viewpoint sequence is decoded, and two viewpoints in the decoded viewpoint sequence are randomly selected to exchange positions. Then, the mutated sequence is re-encoded. According to this rule, all stations are traversed, and the corresponding viewpoint sequences are crossed.
[0047] Update the population. Decode all chromosomes to obtain their cluster information, station sequence information, and viewpoint sequence information. Based on this information, calculate the fitness. Calculate the fitness for all individuals in the population, then sort the individuals in descending order of fitness, and obtain the current best individual. If the current best individual is better than the global best individual, update the global best individual. The fitness is the inverse of the motion cost. The motion cost is calculated as follows:
[0048] Let the station sequence be The number of stations is The viewpoint sequence belonging to the i-th station is The number of viewpoints is The j-th viewpoint is denoted as
[0049] The total inter-viewpoint motion cost is the sum of four terms: the first term is the intra-station viewpoint motion cost, the second term is the inter-station viewpoint motion cost, the third term is the cost from the HOME viewpoint to the starting viewpoint, and the last term is the cost from the final viewpoint to the HOME viewpoint. v (·) is used to calculate the motion cost between two viewpoints. Its formula is:
[0050]
[0051] The inter-station motion cost is:
[0052]
[0053] where s H is the HOME station. s (·) is used to calculate the motion cost between two stations.
[0054] In the calculation of the above motion cost, the motion cost between any two stations and the motion cost between any two viewpoints need to be calculated in advance, which requires the help of kinematics simulation. Specifically, the embodiment uses a detection device containing a mobile robot, a mechanical arm, and a vision sensor, and performs kinematics simulation based on PyBullet to calculate the motion cost.
[0055] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application fall within the scope of protection of the present application.
Claims
1. A station-viewpoint joint clustering and sequence planning method for mobile visual inspection, characterized in that, The method includes the following steps: Step S100: Data encoding and decoding, that is, simultaneously encoding clustering information, station sequence information, and viewpoint sequence information into the gene sequence of the same chromosome; wherein, the clustering information determines the station to which each viewpoint belongs; for each gene, the encoding rule is as follows: the integer part represents the current viewpoint number, and the fractional part represents the station number to which the current viewpoint belongs; Step S200: Population initialization, including: setting the population size, randomly initializing each individual in the population according to the encoding rules; setting the number of iterations, and performing the following steps iteratively; Step S300: Evolutionary strategy update, which controls subsequent crossover and mutation operations through three Boolean variables. These three Boolean variables correspond to three items: clustering, station sequence, and viewpoint sequence, respectively. The evolutionary strategy is updated mainly based on the number of iterations. Step S400: Individual selection, that is, according to the principle that the greater the fitness, the greater the selection probability, pairs of individuals are selected from the population for subsequent crossover operations; the selected individuals are the parents, and the crossover individuals are the offspring. Step S500: Individual crossover, which includes three independent sub-operations, respectively targeting clustering, station sequence, and viewpoint sequence; clustering crossover swaps the fractional parts of corresponding genes to update the station to which the current viewpoint belongs; in one evolutionary process, not all of these operations are executed, but are controlled by the update evolution strategy; Step S600: Individual mutation, which includes three independent sub-operations: cluster-oriented, station sequence-oriented, and viewpoint sequence-oriented mutation sub-operations, which are controlled by the update evolution strategy. Step S700: Population update, which involves decoding all chromosomes to obtain their clustering information, station sequence information, and viewpoint sequence information; based on this information, fitness calculation is performed; fitness is calculated for all individuals in the population, and then individuals are sorted in descending order of fitness, and the current best individual is obtained; if the current best individual is better than the global best individual, the global best individual is updated.
2. The station-viewpoint joint clustering and sequence planning method for mobile visual inspection according to claim 1, characterized in that, In step S100, the length of the chromosome is equal to the number of viewpoints involved in the sorting; for each gene, its encoding rules are as follows: the integer part represents the current viewpoint number, and the fractional part represents the station number to which the current viewpoint belongs; for viewpoints belonging to the same station, their sequence follows the order of their appearance in the chromosome; the station sequence follows the order of their first appearance; therefore, the head segment of the chromosome is used to represent the station sequence.
3. The station-viewpoint joint clustering and sequence planning method for mobile visual inspection according to claim 1, characterized in that, In step S300, the evolutionary strategy is updated primarily based on the number of iterations; If the first variable is TRUE, then cluster crossover and cluster mutation will be performed subsequently; otherwise, they will not be performed. The entire evolution process is divided into three stages according to the number of iterations. In the first stage, the following strategies are performed alternately: (1) optimize only clustering, (2) optimize only station sequence. In the second stage, the following strategies are performed alternately: (1) optimize only clustering, (2) optimize only viewpoint sequence. In the third stage, the following strategies are performed alternately: (1) optimize only clustering, (2) optimize both station sequence and viewpoint sequence, (3) optimize both clustering, station sequence and viewpoint sequence.
4. The station-viewpoint joint clustering and sequence planning method for mobile visual inspection according to claim 1, characterized in that, In step S500, cluster crossover swaps the fractional parts of corresponding genes to update the station to which the current viewpoint belongs. Specifically, when swapping gene pairs at a certain location, it should first be determined whether the viewpoint and the swapped station are reachable. If they are not reachable, the gene at that location will not participate in the swap. For station sequence crossover, the station sequence is first decoded, treated as a crossover operation of the classic traveling salesman problem, and the crossover result is re-encoded to obtain the crossover chromosome. Chromosome segments participating in viewpoint sequence crossover must come from the same station. At this time, the crossover operation of the viewpoint sequence also needs to be decoded and re-encoded. If the viewpoint sets belonging to the same station are inconsistent in two crossover chromosome segments, their intersection needs to be taken first, and then crossover is performed in the above manner. According to this rule, all stations are traversed, and the corresponding viewpoint sequences are crossovered.
5. The station-viewpoint joint clustering and sequence planning method for mobile visual inspection according to claim 1, characterized in that, In step S600, cluster mutation randomly selects a gene from the chromosome; decodes the viewpoint number from the fractional part of the gene; randomly selects a different station from the current station from the set of stations corresponding to the viewpoint and encodes it to update the fractional part of the gene; station sequence mutation first decodes the station sequence and treats it as a mutation operation of the classic traveling salesman problem, that is, randomly selects two stations and swaps their positions to achieve mutation; the mutation result is re-encoded to obtain the mutated chromosome; For viewpoint sequence mutation, the viewpoint sequence at a certain station is decoded, and two viewpoints in the decoded viewpoint sequence are randomly selected and their positions are swapped; then, the mutated sequence is re-encoded; following this rule, all stations are traversed and the corresponding viewpoint sequences are interleaved.