A method, device and equipment for constructing a virtual-reality integrated maintainability test scene

Abstract

The present application relates to a method, device and computer equipment for constructing a virtual-reality fusion maintainability test scene. The method comprises: obtaining a three-dimensional visual video stream of a real maintenance scene of the equipment to be repaired through the binocular lens of AR glasses, and obtaining virtual scene information of a virtual maintenance scene, respectively extracting the first feature information of the physical object of the maintenance object and the second feature information of the virtual model of the maintenance object, performing registration of the physical object of the maintenance object and the virtual maintenance scene through the IPC algorithm, obtaining a transformation matrix of the posture transformation relationship between the physical object of the maintenance object and the virtual maintenance scene, registering the virtual maintenance scene to the real maintenance scene, and completing the construction of a virtual-reality fusion maintainability test scene. The present invention utilizes augmented reality technology to seamlessly integrate physical equipment and virtual cabin environments to form an integrated virtual-reality fusion maintainability test environment, taking into account both accuracy and economy, and has great application prospects.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and computer equipment for constructing a virtual-reality fusion maintainability test scenario for maintainability assessment. Background Art

[0002] Maintainability is an important quality characteristic that reflects whether a product is easy, fast and economical to maintain. In order to ensure that the product has high availability and low life cycle cost, the product must have good maintainability, thereby reducing the requirements for manpower, time and maintenance resources during its maintenance. Therefore, industrial products must undergo sufficient maintainability tests during the development process to verify and evaluate their maintainability to ensure that the specified maintainability requirements are met.

[0003] The traditional method of maintaining performance evaluation based on physical prototypes is overly dependent on physical prototypes, which is costly and sometimes impractical. The method of maintaining performance evaluation using digital prototypes is difficult to accurately evaluate the maintenance force characteristics and maintenance time indicators because accurate human-machine force interaction is very difficult. The existing technology has the problem of poor adaptability. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device and computer equipment for constructing a virtual-reality fusion maintainability test scenario for maintainability assessment, which can enhance the adaptability of maintainability assessment, in response to the above technical problems.

[0005] A method for constructing a virtual-reality fusion maintainability test scenario, the method comprising:

[0006] Acquire a three-dimensional visual video stream of a real maintenance scene of the equipment to be repaired through the binocular lens of the AR glasses; the three-dimensional visual video stream image includes the actual object to be repaired;

[0007] Extracting first feature information of the maintenance object according to the three-dimensional visual video stream image;

[0008] Acquire virtual scene information of a virtual maintenance scene in a preset digital prototype of the equipment to be repaired; the virtual scene information includes a virtual model of the maintenance object;

[0009] Extracting second feature information of the virtual model of the maintenance object according to the virtual scene information;

[0010] According to the first feature information and the second feature information, registering the physical object to be maintained and the virtual maintenance scene by using an IPC algorithm to obtain a transformation matrix of a posture transformation relationship between the physical object to be maintained and the virtual maintenance scene;

[0011] The virtual maintenance scene is registered into the real maintenance scene according to the transformation matrix, so as to complete the construction of the virtual-real fusion maintainability test scene.

[0012] In one embodiment, the method further includes: obtaining maintenance action information of a real person in the virtual-reality fusion maintainability test scene, and synchronously generating maintenance action information of a virtual person in the digital prototype;

[0013] The virtual maintenance evaluation information is obtained according to the virtual human maintenance action information; the virtual maintenance evaluation information includes: virtual maintenance visibility index information, virtual maintenance accessibility index information, virtual maintenance comfort index information and virtual maintenance time information.

[0014] In one embodiment, the method further includes: after obtaining virtual maintenance evaluation information according to the maintenance action information of the virtual person, obtaining real person maintenance evaluation information filled in by the real person after completing the maintenance; the real person maintenance evaluation information is in a data table format, and the real person maintenance evaluation information includes: real person maintenance visibility index information, real person maintenance accessibility index information, real person maintenance comfort index information and real person maintenance time information;

[0015] According to the virtual maintenance evaluation information and the real maintenance evaluation information, after compatibility verification and credibility analysis, weighted fusion is performed according to preset weights to obtain a virtual-real fusion evaluation result of the maintainability of the equipment to be repaired.

[0016] In one of the embodiments, the method further includes: extracting first feature information of the maintenance object by using an ORB feature extraction method according to the three-dimensional visual video stream image.

[0017] In one of the embodiments, the method further includes: extracting second feature information of the virtual model of the maintenance object by using an ORB feature extraction method according to the virtual scene information.

[0018] In one embodiment, the method further includes: obtaining a set of the first feature information K1={k 1i ∈R 3 , i=1,2,…,n} and the set of the second feature information K2={k 2i ∈R 3 ,i=1,2,…,n};

[0019] Sampling the set K1, we get K 10 represents a subset of set K1;

[0020] Search in set K2 and find the 10 The nearest point of each point in , and get the initial correspondence between K1 and K2;

[0021] Remove the wrong corresponding point pairs;

[0022] According to the corresponding relationship, the transformation relationship between K1 and K2 is calculated to minimize the value of the objective function, and the calculated transformation matrix is ​​applied to K 10 , and obtain the updated K after the change 10 ';

[0023] according to Determine whether the iteration is terminated. If d is greater than a preset threshold, continue the iteration. If d is less than the preset threshold or reaches the set number of iterations, the iteration stops, and obtains the transformation matrix of the posture transformation relationship between the physical maintenance object and the virtual maintenance scene.

[0024] In one embodiment, the objective function is:

[0025]

[0026] Among them, R represents the rotation transformation matrix, and T represents the translation transformation matrix.

[0027] A virtual-reality fusion maintainability test scene construction device, the device comprising:

[0028] A real maintenance scene acquisition module is used to acquire a three-dimensional visual video stream of a real maintenance scene of the equipment to be repaired through the binocular lens of the AR glasses; the three-dimensional visual video stream image includes a physical object to be repaired;

[0029] A first feature information extraction module, used to extract first feature information of the maintenance object according to the three-dimensional visual video stream image;

[0030] A virtual maintenance scene acquisition module is used to acquire virtual scene information of a virtual maintenance scene in a preset digital prototype of the equipment to be repaired; the virtual scene information includes a virtual model of a maintenance object;

[0031] A second feature information extraction module, used to extract second feature information of the virtual model of the maintenance object according to the virtual scene information;

[0032] a transformation matrix determination module, configured to perform registration of the physical object to be maintained and the virtual maintenance scene through an IPC algorithm according to the first feature information and the second feature information, and obtain a transformation matrix of a posture transformation relationship between the physical object to be maintained and the virtual maintenance scene;

[0033] The virtual-reality fusion scene construction module is used to register the virtual maintenance scene into the real maintenance scene according to the transformation matrix to complete the construction of the virtual-reality fusion maintainability test scene.

[0034] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Acquire a three-dimensional visual video stream of a real maintenance scene of the equipment to be repaired through the binocular lens of the AR glasses; the three-dimensional visual video stream image includes the actual object to be repaired;

[0036] Extracting first feature information of the maintenance object according to the three-dimensional visual video stream image;

[0037] Acquire virtual scene information of a virtual maintenance scene in a preset digital prototype of the equipment to be repaired; the virtual scene information includes a virtual model of the maintenance object;

[0038] Extracting second feature information of the virtual model of the maintenance object according to the virtual scene information;

[0039] According to the first feature information and the second feature information, registering the physical object to be maintained and the virtual maintenance scene by using an IPC algorithm to obtain a transformation matrix of a posture transformation relationship between the physical object to be maintained and the virtual maintenance scene;

[0040] The virtual maintenance scene is registered into the real maintenance scene according to the transformation matrix, so as to complete the construction of the virtual-real fusion maintainability test scene.

[0041] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0042] Acquire a three-dimensional visual video stream of a real maintenance scene of the equipment to be repaired through the binocular lens of the AR glasses; the three-dimensional visual video stream image includes the actual object to be repaired;

[0043] Extracting first feature information of the maintenance object according to the three-dimensional visual video stream image;

[0044] Acquire virtual scene information of a virtual maintenance scene in a preset digital prototype of the equipment to be repaired; the virtual scene information includes a virtual model of the maintenance object;

[0045] Extracting second feature information of the virtual model of the maintenance object according to the virtual scene information;

[0046] According to the first feature information and the second feature information, registering the physical object to be maintained and the virtual maintenance scene by using an IPC algorithm to obtain a transformation matrix of a posture transformation relationship between the physical object to be maintained and the virtual maintenance scene;

[0047] The virtual maintenance scene is registered into the real maintenance scene according to the transformation matrix, so as to complete the construction of the virtual-real fusion maintainability test scene.

[0048] The above-mentioned virtual-reality fusion maintainability test scene construction method, device, computer equipment and storage medium obtain the three-dimensional visual video stream of the real maintenance scene of the equipment to be repaired through the binocular lens of the AR glasses, and obtain the virtual scene information of the virtual maintenance scene in the preset digital prototype of the equipment to be repaired, respectively extract the first feature information of the real object of the maintenance object and the second feature information of the virtual model of the maintenance object, and align the real object of the maintenance object and the virtual maintenance scene through the IPC algorithm to obtain the transformation matrix of the posture transformation relationship between the real object of the maintenance object and the virtual maintenance scene, register the virtual maintenance scene to the real maintenance scene, and complete the construction of the virtual-reality fusion maintainability test scene. The present invention uses augmented reality technology to seamlessly integrate the physical equipment and the virtual cabin environment to form an integrated virtual-reality fusion maintainability test environment, which can not only truly show the maintenance dynamic characteristics of the equipment, but also show the influence of the cabin environment on the maintenance space and maintenance process, realize reliable maintainability evaluation, and at the same time reduce the hardware scale, taking into account both accuracy and economy, and has great application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a method for constructing a virtual-reality fusion maintainability test scenario in an embodiment;

[0050] Figure 2 The figure is a physical object of a YN92 diesel engine in a specific embodiment, wherein a is an axonometric view of the physical object of the YN92 diesel engine, and b is a left view of the physical object of the YN92 diesel engine;

[0051] Figure 3 This is a schematic diagram of a virtual maintenance scene of a ship auxiliary engine room in a specific embodiment;

[0052] Figure 4 A schematic diagram of a diesel engine feature extraction result using the ORB method in a specific embodiment;

[0053] Figure 5 Schematic diagrams showing the comparison of models after some parts are removed from the equipment model in a specific embodiment, wherein a is the original model, b is the model 2 after the left cylinder head is removed, c is the model 3 after the right cylinder head is removed, d is the model 4 after the small motor is removed, e is the model 5 after the air filter cover is removed, and f is the model 6 after the starter motor is removed;

[0054] Figure 6 A virtual-real fusion ship engine maintenance scene diagram obtained in a specific embodiment;

[0055] Figure 7A flowchart of the operation steps of starting a motor maintenance process in a specific embodiment;

[0056] Figure 8 It is an overall flow chart of maintainability test evaluation based on virtual-reality fusion maintenance scenario in another specific embodiment;

[0057] Fig. 9 A structural block diagram of a device for constructing a virtual-reality fusion maintainability test scenario in an embodiment;

[0058] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] The method for constructing a virtual-reality fusion maintainability test scene provided in the present application can be applied in the following application environments. Among them, the terminal executes a method for constructing a virtual-reality fusion maintainability test scene, including obtaining a three-dimensional visual video stream of a real maintenance scene of the equipment to be repaired through the binocular lens of the AR glasses, and obtaining the virtual scene information of the virtual maintenance scene in the preset digital prototype of the equipment to be repaired, respectively extracting the first feature information of the physical object to be repaired and the second feature information of the virtual model of the maintenance object, and aligning the physical object to be repaired and the virtual maintenance scene through the IPC algorithm, obtaining the transformation matrix of the posture transformation relationship between the physical object to be repaired and the virtual maintenance scene, registering the virtual maintenance scene to the real maintenance scene, and completing the construction of the virtual-reality fusion maintainability test scene. Among them, the terminal can be, but is not limited to, various personal computers, laptops, tablet computers, and portable wearable devices.

[0061] In one embodiment, Figure 1 As shown, a method for constructing a virtual-reality fusion maintainability test scenario is provided, comprising the following steps:

[0062] Step 102: Acquire a three-dimensional visual video stream of a real maintenance scene of the equipment to be repaired through the binocular lens of the AR glasses.

[0063] Step 104: extract first feature information of the maintenance object based on the three-dimensional visual video stream image.

[0064] After the camera internal parameters are calibrated, the features of the video image are extracted.

[0065] Step 106: Acquire virtual scene information of a virtual maintenance scene in a preset digital prototype of the equipment to be repaired.

[0066] During the virtual-reality integrated maintainability test of a product, a complete set of digital prototypes of the product is usually available as the information basis for the test. The digital prototype reflects the positional relationship between the physical product and the surrounding environment.

[0067] Step 108: extracting second characteristic information of the virtual model of the maintenance object according to the virtual scene information.

[0068] Step 110: perform registration of the physical maintenance object and the virtual maintenance scene through an IPC algorithm according to the first feature information and the second feature information, and obtain a transformation matrix of the position and posture transformation relationship between the physical maintenance object and the virtual maintenance scene.

[0069] In order to superimpose the virtual maintenance environment model on the periphery of the physical product object and make it sure that it is a part of the maintenance environment, the physical product must be identified and the virtual world and the physical world must be fully aligned.

[0070] Step 112, registering the virtual maintenance scene into the real maintenance scene according to the transformation matrix, and completing the construction of the virtual-reality fusion maintainability test scene.

[0071] In the above-mentioned virtual-reality fusion maintainability test scene construction method, the three-dimensional visual video stream of the real maintenance scene of the equipment to be repaired is obtained through the binocular lens of the AR glasses, and the virtual scene information of the virtual maintenance scene in the preset digital prototype of the equipment to be repaired is obtained, and the first feature information of the physical object of the maintenance object and the second feature information of the virtual model of the maintenance object are respectively extracted, and the physical object of the maintenance object and the virtual maintenance scene are registered by the IPC algorithm to obtain the transformation matrix of the posture transformation relationship between the physical object of the maintenance object and the virtual maintenance scene, and the virtual maintenance scene is registered to the real maintenance scene to complete the construction of the virtual-reality fusion maintainability test scene. The present invention uses augmented reality technology to seamlessly integrate physical equipment and virtual cabin environment to form an integrated virtual-reality fusion maintainability test environment, which can not only truly show the maintenance dynamic characteristics of the equipment, but also show the influence of the cabin environment on the maintenance space and maintenance process, realize reliable maintainability evaluation, and at the same time reduce the hardware scale, taking into account both accuracy and economy, and has great application prospects.

[0072] In one of the embodiments, it also includes: synchronously generating virtual human maintenance action information in the digital prototype by acquiring real human maintenance action information in a virtual-reality fusion maintainability test scenario; acquiring virtual maintenance evaluation information based on the virtual human maintenance action information; the virtual maintenance evaluation information includes: virtual maintenance visibility index information, virtual maintenance accessibility index information, virtual maintenance comfort index information and virtual maintenance time information.

[0073] After obtaining the virtual maintenance evaluation information based on the virtual person's maintenance action information, obtain the real person's maintenance evaluation information filled out by the real person after completing the maintenance; the real person's maintenance evaluation information is in a data table format, and the real person's maintenance evaluation information includes: real person maintenance visibility index information, real person maintenance accessibility index information, real person maintenance comfort index information and real person maintenance time information; based on the virtual maintenance evaluation information and the real person's maintenance evaluation information, after compatibility testing and credibility analysis, weighted fusion is performed according to preset weights to obtain a virtual-reality fusion evaluation result of the maintainability of the equipment to be repaired.

[0074] In theory, the evaluation results of real-person experience are more credible than the virtual evaluation results of software. This embodiment mainly uses real-person experience and virtual evaluation results as an auxiliary to perform weighted average calculation on virtual and real data samples to achieve virtual and real information fusion of maintainability evaluation results.

[0075] Next, we introduce some evaluation methods of the main indicators:

[0076] (1) Maintenance visibility

[0077] Maintenance visibility refers to the visibility of the maintenance part during maintenance. Good visibility of the maintenance part makes it easier for maintenance personnel to observe the use of components, reduce the difficulty of maintenance, and facilitate maintenance personnel to complete maintenance tasks. On the contrary, if the maintenance part is invisible or difficult to see, it will greatly increase the difficulty of maintenance and increase the fatigue intensity of maintenance personnel, and even affect the completion of maintenance tasks. The corresponding values ​​of the subjective evaluation of visibility indicators are shown in Table 1.

[0078] Table 1 Evaluation values ​​and standards of visibility indicators

[0079] Visibility Index Evaluation Criteria Visibility Index Value Both eyes are fully visible 0.8—1.0 Partially visible in both eyes 0.7 Fully visible to one eye 0.5 Partially visible in one eye 0.3 Invisible 0.1

[0080] (2) Maintenance accessibility

[0081] Maintenance accessibility refers to the ease with which maintenance personnel can approach the maintenance parts during equipment maintenance, which can reflect the impact of the working space, maintenance passage, parts layout, etc. on maintenance operations. The quality of accessibility directly affects equipment inspection and maintenance, so full attention should be paid to maintenance accessibility. The corresponding values ​​of the subjective evaluation of accessibility indicators are shown in Table 2.

[0082] Table 2 Evaluation values ​​and standards of accessibility indicators

[0083]

[0084] (3) Operation space

[0085] The operating space is related to the feasibility and convenience of maintenance, which is mainly ensured by the installation and layout of equipment or parts. The equipment must have a certain maintenance operation space to facilitate maintenance operations. Similarly, the corresponding values ​​of the subjective evaluation of the operating space index are shown in Table 3.

[0086] Table 3 Evaluation levels and standards of operating space indicators

[0087]

[0088] (4) Average time for corrective maintenance

[0089] The mean corrective maintenance time is the average of the actual repair time required to eliminate a fault. It is a basic parameter of equipment maintainability, also known as the mean repair time, and is usually measured using the cumulative time estimation method.

[0090] The cumulative model uses basic maintenance operations to synthesize the maintenance activity time T mnj ,The maintenance activity time is synthesized into the average repair time R of each main replaceable unit under each fault detection and isolation output nj (n represents the nth unit, j represents the jth output), that is:

[0091]

[0092] Where: M nj The number of activities performed to correct the fault after the nth RI fails and is detected by the jth FD&I. This includes various maintenance activities, i.e., preparation, isolation, etc. This may include actions for other RIs detected by the jth fault isolation result (e.g., using alternating replacement to identify the jth RI)

[0093] T mnj It is the average time to perform the mth troubleshooting maintenance activity on the nth RI detected by the jth FD&I output.

[0094] Then the mean model is used to calculate the average repair time, that is:

[0095]

[0096] Where: J is the total number of various FD&I outputs; λ nj is the failure rate of the nth RI that can be detected at the jth FD&I output; R nj is the fault repair time of the nth RI detected at the jth FD&I output.

[0097] The mean time to repair is calculated as:

[0098]

[0099] In one of the embodiments, it also includes: extracting first feature information of the physical object of the maintenance object through the ORB feature extraction method according to the three-dimensional visual video stream image; and extracting second feature information of the virtual model of the maintenance object through the ORB feature extraction method.

[0100] In one embodiment, the process of extracting features of the image of the maintainability test object based on ORB is as follows:

[0101] FAST feature points are not directional, and the directional parameters are determined by obtaining the centroid of the feature point neighborhood. The neighborhood moment is:

[0102]

[0103] Where I(x,y) is the grayscale value at point (x,y), x,y∈[-r,r], r is the radius of the circle, p and q are non-negative integers, when p is 1 and q is 0, the value of I in the x direction can be obtained: x , when p is 0 and q is 1, the value of I in the y direction can be obtained y , so the coordinates of the image center of gravity C are:

[0104]

[0105] The angle between the feature point and the center of gravity is defined as the direction of the FAST feature point:

[0106]

[0107] ORB extracts the BRIEF descriptor based on the direction parameter obtained from the above formula. However, due to environmental factors and the introduction of noise, the direction of the feature point will change, and the correlation of random pixel blocks will be relatively large, thereby reducing the discriminability of the descriptor. ORB uses a greedy algorithm to find random pixel blocks with low correlation, and generally selects 256 pixel blocks with the lowest correlation to form a 256-bit feature descriptor. Remember two descriptors:

[0108] K1=x0x1…x 255 ,K2=y0y1…y 255

[0109] At present, many local features such as SIFT, SURF, ORB, BRISK, FREAK, etc. are widely used in image matching and object recognition. Since the objects of the maintainability test process are usually mechanical products, their surfaces sometimes lack rich texture features. Considering the stability and rapidity of feature point extraction and matching, this embodiment uses ORB local features. ORB local features use FAST as feature point detector, improved BRIEF as feature descriptor, and BF pattern matching algorithm for feature descriptor matching.

[0110] In one embodiment, the process of matching and identifying the physical device features and registering the virtual environment is as follows:

[0111] The ORB feature sets are extracted from the real maintainability test object and the virtual maintenance environment model respectively, and the corresponding feature descriptors K1 and K2 are obtained. The XOR sum of the Hamming distances represents the similarity of the two ORB feature descriptors:

[0112]

[0113] The smaller D(K1, K2) is, the higher the similarity is, and the greater the probability that the two describe the same feature. On the contrary, the lower the similarity is, the more likely it is that the two do not describe the same feature.

[0114] Use the BF matcher to obtain all possible matching feature pairs, assuming that the minimum Hamming distance of the feature pair is MIN_DIST. In order to select the best matching pair and improve the running efficiency, select a suitable threshold and select the matching pairs less than the threshold for the subsequent camera pose estimation. The threshold cannot be selected too small, which will affect the final effect. It is necessary to select the best threshold by experimenting with the image frames.

[0115] Given point k in K1 1i , find the distance k from K2 1i The point k with the shortest Euclidean distance 2i , and k 1i and k 2i The transformation matrix is ​​obtained as the corresponding point. Through continuous iteration, the following formula is minimized, then the iteration is terminated, and finally the optimal transformation matrix is ​​obtained to make the two coincide.

[0116]

[0117] Where R represents the rotation transformation matrix and represents the translation transformation matrix.

[0118] The essence of the ICP algorithm is to calculate the transformation matrix between the feature sets, and minimize the registration error between the two by rotation and translation to achieve the best registration effect. Assume that the two feature point sets K1 = {k 1i ∈R 3 ,i=1,2,…,n} and K2={k 2i ∈R 3 ,i=1,2,…,n}, the following is an introduction to the registration process of the two using the ICP algorithm:

[0119] (1) Sampling the set K1, K 10 represents a subset of the set K1;

[0120] (2) Search in set K2 and find the 10 The nearest point of each point in , and get the initial correspondence between K1 and K2;

[0121] (3) Use algorithms or limiting conditions to remove incorrect corresponding point pairs;

[0122] (4) Calculate the transformation relationship between the two according to the corresponding relationship in step (2) to minimize the value of the objective function, and apply the calculated transformation matrix to K 10 , and the new K after the change is obtained 10 ;

[0123] (5) Based on To determine whether the iteration is terminated, if d is greater than the preset threshold, return to (2) to continue the iteration; if d is less than the preset threshold or reaches the set number of iterations, the iteration stops.

[0124] By obtaining the transformation matrix through the above steps, the posture transformation relationship between the physical equipment and the virtual maintainability test environment can be obtained, and then virtual registration can be performed to complete the construction of the virtual-reality fusion maintainability test environment.

[0125] In a specific embodiment, the auxiliary engine room of a certain ship is used as a case to carry out test verification to verify the correctness and applicability of the virtual-reality fusion maintainability test evaluation method studied in this paper. The auxiliary engine room is powered by a diesel engine for power generation. The diesel engine consists of a crank-connecting rod mechanism, a valve structure, a fuel system, a lubrication system, a cooling system, a starting system, etc. The engine needs to replace consumable parts such as fuel filters and air filters, and the cylinder and starter motor have a certain failure rate, so they need to be well designed for maintainability to ensure rapid maintenance at the crew level.

[0126] In the ship cabin environment, the equipment maintenance process is somewhat complicated, and other equipment around the equipment and peripheral pipelines and cables easily lead to insufficient accessibility and operation space for the maintenance object. Therefore, during the engine maintainability test, it is necessary to simulate the actual cabin maintenance scene and maintenance space, and fully consider the impact of various operational obstacles on maintainability, so as to obtain more accurate maintainability test results.

[0127] Since the establishment of 1:1 full-scale maintainability test conditions is very costly and time-consuming, the virtual-reality fusion maintainability test evaluation method studied in this paper is adopted to simulate the complete test scenario realistically using a small number of physical equipment and a large number of virtual environments. During the test, the test conditions include the YN92 real diesel engine and the complete digital model of the auxiliary engine compartment, such as Figure 2 , Figure 3 As shown, Figure 2a is the real object of YN92 diesel engine (axonometric view), Figure 2 b is the real object of YN92 diesel engine (left view), Figure 3 This is a virtual maintenance scene for the auxiliary engine room of a ship. Next, we will take the maintenance and replacement of the starter motor as an example for verification.

[0128] (1) In order to construct a more realistic virtual-reality fusion maintenance scene, it is necessary to consider the impact of multiple factors on the registration accuracy of the virtual environment. Among them, the feature extraction method is an important factor affecting the registration accuracy.

[0129] First, the diesel engine is feature extracted and identified. Different feature extraction methods have different feature extraction results. The feature extraction results of the diesel engine are obtained by using the ORB method. Figure 4 shown.

[0130] (2) To ensure the accuracy of virtual environment registration, we must also consider the matching effect when the physical object and its corresponding model are not completely consistent. Modify the model of the physical object in the CAD environment, and calculate and analyze the matching accuracy based on feature extraction. Figure 5 This is a comparison display of the equipment model after some parts are removed. a is the original model, b is the model 2 with the left cylinder head removed (the size changes by about 5%), c is the model 3 with the right cylinder head removed (the size changes by about 10%), d is the model 4 with the small motor removed (the size changes by about 15%), e is the model 5 with the air filter cover removed (the size changes by about 20%), and f is the model 6 with the starter motor removed (the size changes by about 25%).

[0131] The following table shows the matching results after removing some parts from the equipment model as shown in Table 5.

[0132] Table 5 Matching results of different models

[0133]

[0134]

[0135] As can be seen from Table 5, the more similar the model is to the real object, the higher the matching accuracy. When registering in a virtual environment, we must ensure the registration accuracy and prevent registration failure caused by inconsistency between the model and the real object, so we must choose a suitable matching threshold. After comprehensive analysis, we can see that threshold 2 is more appropriate, which can ensure a certain registration accuracy and avoid registration failure.

[0136] The feature extraction algorithm and feature matching recognition algorithm are injected into AR glasses. The three-dimensional visual information of the physical equipment is obtained through the binocular lens of the glasses. The features are then extracted and matched one by one with the virtual model. The virtual-real fusion ship cabin maintenance scene is obtained as follows: Figure 6 shown.

[0137] Next, based on the established virtual-reality fusion maintainability test scenario for the YN92 real diesel engine, a maintainability operation test for replacing the starter motor was carried out. The test personnel wore AR glasses to perform the maintainability test and obtain basic test data. The maintenance process operation steps are as follows: Figure 7 shown.

[0138] A total of five groups of tests were conducted. In each group of tests, maintenance operations were performed in three scenarios: real environment, virtual-reality fusion, and no surrounding environment. The maintainability (visibility, accessibility, operating space, and operating time) results of the maintenance tests were obtained. The obtained maintainability test data are shown in Table 6.

[0139] Table 6 Statistics of some maintainability test data of starting motor

[0140]

[0141]

[0142] It is calculated that the average error of the three maintainability qualitative indicators in the virtual-reality fusion scenario is 9.90%, and the average error of the maintenance time is 3.25%. The error of the three maintainability qualitative indicators in the scenario without surrounding environment is 22.29%, and the average error of the maintenance time is 9.45%. It can be found that the relative error of the virtual-reality fusion maintainability evaluation results is significantly reduced compared with the results of the test without surrounding environment. Because the maintenance personnel can feel the existence of the surrounding cabin equipment through vision in the virtual-reality fusion maintenance test, in order to avoid colliding with the virtual cabin equipment during maintenance, the bending angle of the arm will be smaller, the movement range will not be large, and the posture of the maintenance personnel should be adjusted accordingly, which is closer to the real maintenance situation, so the error is smaller and closer to the real maintenance. The main reason for the error in the virtual-reality fusion maintenance test is the influence of the psychological factors of the maintenance personnel. It is also possible that the maintenance personnel cross the virtual environment during maintenance and fail to correct the action in time. From the above, it can be seen that the virtual-reality fusion maintainability evaluation method takes into account the influence of the surrounding environment, which significantly enhances the accuracy and credibility of the test evaluation results.

[0143] In another specific embodiment, Figure 8 As shown, the overall flow chart of maintainability test evaluation based on virtual-reality fusion maintenance scenario is provided, including:

[0144] According to the real camera movement, the test object image is acquired, the target features are extracted, and feature recognition and matching are performed with the virtual maintenance environment prototype object image to obtain posture data. The virtual camera movement is controlled according to the posture data to obtain the transformed virtual maintenance environment. The real maintainability test object and the transformed virtual maintenance environment are fused and displayed to obtain a virtual-reality fusion maintainability test scene. According to the maintainability test conducted in the virtual-reality fusion maintainability test scene, maintainability test data is obtained to obtain the maintainability evaluation result.

[0145] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0146] In one embodiment, Fig. 9 A virtual-reality fusion maintainability test scene construction device is provided, comprising: a real maintenance scene acquisition module 902, a first feature information extraction module 904, a virtual maintenance scene acquisition module 906, a second feature information extraction module 908, a transformation matrix determination module 910 and a virtual-reality fusion scene construction module 912, wherein:

[0147] The real maintenance scene acquisition module 902 is used to acquire a three-dimensional visual video stream of a real maintenance scene of the equipment to be repaired through the binocular lens of the AR glasses; the three-dimensional visual video stream image includes the physical object to be repaired;

[0148] A first feature information extraction module 904, used to extract first feature information of the maintenance object according to the three-dimensional visual video stream image;

[0149] The virtual maintenance scene acquisition module 906 is used to acquire virtual scene information of a virtual maintenance scene in a preset digital prototype of the equipment to be maintained; the virtual scene information includes a virtual model of the maintenance object;

[0150] A second feature information extraction module 908, used to extract second feature information of the virtual model of the maintenance object according to the virtual scene information;

[0151] The transformation matrix determination module 910 is used to perform registration of the physical maintenance object and the virtual maintenance scene through the IPC algorithm according to the first feature information and the second feature information, and obtain a transformation matrix of the posture transformation relationship between the physical maintenance object and the virtual maintenance scene;

[0152] The virtual-reality fusion scene construction module 912 is used to register the virtual maintenance scene into the real maintenance scene according to the transformation matrix, so as to complete the construction of the virtual-reality fusion maintainability test scene.

[0153] The virtual-reality fusion scene construction module 912 is also used to synchronously generate virtual human maintenance action information in the digital prototype by acquiring real human maintenance action information in the virtual-reality fusion maintainability test scene; and to acquire virtual maintenance evaluation information based on the virtual human maintenance action information; the virtual maintenance evaluation information includes: virtual maintenance visibility index information, virtual maintenance accessibility index information, virtual maintenance comfort index information and virtual maintenance time information.

[0154] The virtual-reality fusion scene construction module 912 is also used to obtain real-person maintenance evaluation information filled out after a real person completes the maintenance; the real-person maintenance evaluation information is in a data table format, and the real-person maintenance evaluation information includes: real-person maintenance visibility index information, real-person maintenance accessibility index information, real-person maintenance comfort index information and real-person maintenance time information; based on the virtual maintenance evaluation information and the real-person maintenance evaluation information, after compatibility testing and credibility analysis, weighted fusion is performed according to preset weights to obtain a virtual-reality fusion evaluation result of the maintainability of the equipment to be repaired.

[0155] The first feature information extraction module 904 is further used to extract the first feature information of the maintenance object through the ORB feature extraction method according to the three-dimensional visual video stream image.

[0156] The second feature information extraction module 908 is further used to extract the second feature information of the virtual model of the maintenance object by using the ORB feature extraction method according to the virtual scene information.

[0157] The transformation matrix determination module 910 is also used to obtain a set of first feature information K1={k 1i ∈R 3 , i=1,2,…,n} and the set of the second feature information K2={k 2i ∈R 3 ,i=1,2,…,n};

[0158] Sampling the set K1, we get K 10 represents a subset of set K1;

[0159] Search in set K2 and find the 10The nearest point of each point in , and get the initial correspondence between K1 and K2;

[0160] Remove the wrong corresponding point pairs;

[0161] According to the corresponding relationship, the transformation relationship between K1 and K2 is calculated to minimize the value of the objective function, and the calculated transformation matrix is ​​applied to K 10 , and obtain the updated K after the change 10 ';

[0162] according to Determine whether the iteration is terminated. If d is greater than the preset threshold, the iteration continues. If d is less than the preset threshold or the set number of iterations is reached, the iteration stops and the transformation matrix of the posture transformation relationship between the real maintenance object and the virtual maintenance scene is obtained.

[0163] For the specific definition of the virtual-reality fusion maintainability test scene construction device, please refer to the definition of the virtual-reality fusion maintainability test scene construction method mentioned above, which will not be repeated here. Each module in the above-mentioned virtual-reality fusion maintainability test scene construction device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0164] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for constructing a virtual-reality fusion maintainability test scenario is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0165] Those skilled in the art will understand that Fig.10The structure shown is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0166] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0167] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0168] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0169] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for constructing a virtual-reality fusion maintainability test scenario, characterized in that: The method comprises: Acquire a three-dimensional visual video stream of a real maintenance scene of the equipment to be repaired through the binocular lens of the AR glasses; the three-dimensional visual video stream image includes the actual object to be repaired; Extracting first feature information of the maintenance object according to the three-dimensional visual video stream image; Acquire virtual scene information of a virtual maintenance scene in a preset digital prototype of the equipment to be repaired; the virtual scene information includes a virtual model of the maintenance object; Extracting second feature information of the virtual model of the maintenance object according to the virtual scene information; According to the first feature information and the second feature information, registering the physical object to be maintained and the virtual maintenance scene by using an IPC algorithm to obtain a transformation matrix of a posture transformation relationship between the physical object to be maintained and the virtual maintenance scene; The virtual maintenance scene is registered into the real maintenance scene according to the transformation matrix, so as to complete the construction of the virtual-real fusion maintainability test scene.

2. The method according to claim 1, characterized in that After completing the construction of the virtual-reality fusion maintainability test scenario, it also includes: By acquiring maintenance action information of a real person in the virtual-reality fusion maintainability test scene, maintenance action information of a virtual person is synchronously generated in the digital prototype; The virtual maintenance evaluation information is obtained according to the virtual human maintenance action information; the virtual maintenance evaluation information includes: virtual maintenance visibility index information, virtual maintenance accessibility index information, virtual maintenance comfort index information and virtual maintenance time information.

3. The method according to claim 2, characterized in that After obtaining virtual maintenance evaluation information according to the virtual human maintenance action information, the method further includes: Acquire real person maintenance evaluation information filled in by a real person after completing maintenance; the real person maintenance evaluation information is in a data table format, and the real person maintenance evaluation information includes: real person maintenance visibility index information, real person maintenance accessibility index information, real person maintenance comfort index information and real person maintenance time information; According to the virtual maintenance evaluation information and the real maintenance evaluation information, after compatibility verification and credibility analysis, weighted fusion is performed according to preset weights to obtain a virtual-real fusion evaluation result of the maintainability of the equipment to be repaired.

4. The method according to claim 1, characterized in that: Extracting first feature information of the maintenance object according to the three-dimensional visual video stream image includes: According to the three-dimensional visual video stream image, first feature information of the maintenance object is extracted by using an ORB feature extraction method.

5. The method according to claim 4, characterized in that Extracting second feature information of the virtual model of the maintenance object according to the virtual scene information includes: According to the virtual scene information, second feature information of the virtual model of the maintenance object is extracted by using an ORB feature extraction method.

6. The method according to claim 5, characterized in that According to the first feature information and the second feature information, the physical object to be maintained and the virtual maintenance scene are registered by an IPC algorithm to obtain a transformation matrix of a posture transformation relationship between the physical object to be maintained and the virtual maintenance scene, including: Get the first feature information set K1 = {k 1i ∈R 3 , i=1,2,…,n} and the set of the second feature information K2={k 2i ∈R 3 ,i=1,2,…,n}; Sampling the set K1, we get K 10 represents a subset of set K1; Search in set K2 and find 10 The nearest point of each point in , and get the initial correspondence between K1 and K2; Remove the wrong corresponding point pairs; According to the corresponding relationship, the transformation relationship between K1 and K2 is calculated to minimize the value of the objective function, and the calculated transformation matrix is ​​applied to K 10 , and obtain the updated K after the change 10 '; according to Determine whether the iteration is terminated. If d is greater than a preset threshold, continue the iteration. If d is less than the preset threshold or reaches the set number of iterations, the iteration stops, and obtains the transformation matrix of the posture transformation relationship between the physical maintenance object and the virtual maintenance scene.

7. The method according to claim 6, characterized in that The objective function is: Among them, R represents the rotation transformation matrix, and T represents the translation transformation matrix.

8. A virtual-reality fusion maintainability test scene construction device, characterized in that: The device comprises: A real maintenance scene acquisition module is used to acquire a three-dimensional visual video stream of a real maintenance scene of the equipment to be repaired through the binocular lens of the AR glasses; the three-dimensional visual video stream image includes a physical object to be repaired; A first feature information extraction module, used to extract first feature information of the maintenance object according to the three-dimensional visual video stream image; A virtual maintenance scene acquisition module is used to acquire virtual scene information of a virtual maintenance scene in a preset digital prototype of the equipment to be repaired; the virtual scene information includes a virtual model of a maintenance object; A second feature information extraction module, used to extract second feature information of the virtual model of the maintenance object according to the virtual scene information; a transformation matrix determination module, configured to perform registration of the physical object to be maintained and the virtual maintenance scene through an IPC algorithm according to the first feature information and the second feature information, and obtain a transformation matrix of a posture transformation relationship between the physical object to be maintained and the virtual maintenance scene; The virtual-reality fusion scene construction module is used to register the virtual maintenance scene into the real maintenance scene according to the transformation matrix to complete the construction of the virtual-reality fusion maintainability test scene.

9. The device according to claim 8, characterized in that The first feature information extraction module is also used to extract the first feature information of the maintenance object through the ORB feature extraction method according to the three-dimensional visual video stream image.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images