Auxiliary evaluation method, device and equipment of transportation device console and medium

Abstract

The invention relates to the technical field of computer assistance, and discloses an auxiliary evaluation method, device, equipment and medium for a transportation device console, applied to a computer-aided evaluation system, the computer-aided evaluation system comprises a machine vision system, a mechanical arm and a background computer, and the machine vision system and the mechanical arm are in communication connection with the background computer. The background computer is in communication connection with the console; the method comprises the following steps: determining a preset navigation task and determining an operation instruction; and mechanical load index data of the mechanical arm and visual load index data of the machine vision system when the control instruction is executed are obtained, comprehensive evaluation is carried out, and evaluation data on the performance of the preset requirement are obtained. The beneficial effects of the invention lie in that the mechanical arm is utilized to simulate the two hands of the operator, the machine vision system is utilized to simulate the two eyes of the operator, the quantitative evaluation of the preset required performance of the console is realized according to the feedback data of the mechanical arm and the machine vision system, and the accuracy of the evaluation result is improved.

Description

Technical Field

[0001] This application relates to the field of computer-aided technology, and particularly to an auxiliary evaluation method, device, equipment and medium for a console of a transportation device. Background Art

[0002] In common transportation devices, there is usually a dedicated console for operating the transportation device. The transportation device includes vehicles, ships and aircraft. Exemplarily, for a ship, during the navigation process of the ship, the driver usually needs to operate the ship console for a long time and continuously maintain a high degree of accuracy and efficiency to cope with complex navigation environments and changing navigation requirements. The ergonomic performance of the ship console directly affects the driver's operation experience. A reasonable ergonomic design can reduce the fatigue level of the driver during the operation process, enabling the driver to maintain the best state. On the contrary, if the ergonomic design of the ship console is unreasonable, it may cause problems such as rapid physical fatigue or inattention of the driver, increasing the risk of operation errors, and thus posing a safety hazard to the ship navigation process. Therefore, it is crucial to evaluate the ergonomic performance of the transportation device console before actual use.

[0003] In related technologies, the preset requirement performance of the transportation device console is usually evaluated using subjective human data, and the objectivity and accuracy of related technologies still need to be improved. Summary of the Invention

[0004] This application provides an auxiliary evaluation method, device, equipment and medium for a transportation device console, which uses a robotic arm to simulate the hands of an operator and a machine vision system to simulate the eyes of an operator, and realizes a quantitative evaluation of the preset requirement performance of the transportation device console according to the feedback data of the robotic arm and the machine vision system, improving the accuracy of the evaluation result.

[0005] To achieve the above object, the main technical solutions adopted in this application include: In a first aspect, an embodiment of this application provides an auxiliary evaluation method for a transportation device console, which is applied to a computer-aided evaluation system. The computer-aided evaluation system includes a machine vision system, a robotic arm and a background computer; the console is used to control a simulated transportation device during the evaluation process. The console is within the shooting angle of view of the machine vision system and within the control range of the robotic arm. The machine vision system and the robotic arm are communicatively connected to the background computer, and the background computer is communicatively connected to the console; the method includes: Determine a preset navigation task for evaluating the preset requirement performance of the console; Before executing the preset navigation task, obtain the navigation state information of the simulation transport device and the operation state information of the console through the machine vision system; and plan a navigation instruction according to the navigation state information, the operation state information, and the preset navigation task, so as to obtain an operation instruction indicating the manipulator to control the console. Obtain the mechanical load index data of the manipulator during the execution of the operation instruction, and analyze the visual display state of the console through the machine vision system to obtain visual load index data; wherein, the mechanical load index data is used to describe the load condition of the operator's arm when the console is actually controlled, and the visual load index data is used to describe the load condition of the operator's eyes when the console is actually controlled. Conduct a comprehensive evaluation according to the mechanical load index data and the visual load index data to obtain the evaluation data of the console on the preset required performance.

[0006] The auxiliary evaluation method for the transport device console proposed in the embodiment of the present application is applied to a computer-aided evaluation system including a machine vision system, a manipulator, and a background computer. The background computer is used to control the machine vision system and the manipulator to execute the preset navigation task, and receive the feedback data of the console, the machine vision system, and the manipulator. In the present application, first, according to the preset navigation task, use the machine vision system to obtain the navigation state information of the simulation transport device and the operation state information of the console, so as to obtain an operation instruction indicating the manipulator to perform the control; secondly, according to the mechanical load index data fed back by the manipulator and the visual load index data fed back by the machine vision system, conduct a comprehensive evaluation of the preset required performance of the console to obtain the evaluation data. Compared with the related technology, the present application uses the machine vision system to simulate the operator's eyes and the manipulator to simulate the operator's hands, which not only makes the evaluation conditions controllable and stable, improves the standardization degree of the evaluation process, but also can set quantitative indicators for the preset required performance of the console according to the feedback data in the evaluation process of the machine vision system and the manipulator for quantitative analysis, improving the accuracy of the evaluation results.

[0007] Optionally, the transport device is a ship, the simulation transport device is a simulation ship, and the console is a ship console; the mechanical load index data is obtained through the following method: Obtain the control torque data of the manipulator during the execution of the operation instruction. Count the number of times the manipulator controls the same control component on the ship console during the execution of the operation instruction to obtain the component repetition frequency. Count the number of times the manipulator performs the same operation during the execution of the operation instruction to obtain the repeated control data. The mechanical load index data is obtained based on the manipulation torque data, the component repetition frequency, and the repeated manipulation data.

[0008] Optionally, the visual load index data is obtained by the following method: Performing a field of view analysis on the ship console according to the visual display state to obtain field of view data; Performing a visual feature analysis on the visual display state to obtain visual feature data; wherein, the visual feature data includes data obtained by performing color analysis and brightness analysis on the visual display state; Comparing the manipulation state information obtained by the machine vision system with the actual manipulation state information of the ship console, and obtaining error recognition data according to the difference between the manipulation state information and the actual manipulation state information; The visual load index data is obtained based on the field of view data, the visual feature data, and the error recognition data.

[0009] Optionally, the ship console includes a display screen and manipulation components, and the visual display state includes the display screen display state and the manipulation component display state; the visual display state is obtained by the following method: Performing information recognition on the display screen display state to obtain the navigation state information of the simulated ship; Performing manipulation component state recognition on the manipulation component display state to obtain the manipulation state information of the ship console.

[0010] Optionally, the navigation instruction planning is performed according to the navigation state information, the manipulation state information, and the preset navigation task to obtain a manipulation instruction indicating the manipulator to manipulate the ship console, including: Performing navigation instruction planning on the simulated ship according to the navigation state information and the navigation target of the simulated ship in the preset navigation task to obtain a navigation control instruction for controlling the simulated ship; Comparing the navigation control instruction with the manipulation state information of the ship console, determining the relative adjustment amount of each manipulation component on the ship console, and obtaining the manipulation instruction.

[0011] Optionally, before obtaining the mechanical load index data of the manipulator during the execution of the manipulation instruction, it includes: Performing a spatial path planning on the manipulator according to the manipulation instruction to obtain a spatial path for controlling the manipulator.

[0012] Optionally, the performing a spatial path planning on the manipulator according to the manipulation instruction to obtain a spatial path for controlling the manipulator includes: Determine a number of path points in the space where the ship console is located according to the external dimensions, component positions, and relative adjustment amounts of the manipulation components in the manipulation instructions; Perform quaternion interpolation between the number of path points to obtain a space path passing through the path points.

[0013] In a second aspect, an auxiliary evaluation device for a transport device console provided by an embodiment of the present application is applied to a computer-aided evaluation system. The computer-aided evaluation system includes a machine vision system, a robotic arm, and a background computer; the console is used to control a simulated transport device during the evaluation process. The console is within the shooting perspective of the machine vision system and within the control range of the robotic arm. The machine vision system and the robotic arm are communicatively connected to the background computer, and the background computer is communicatively connected to the console; the device includes: A preset task determination module for determining a preset navigation task for evaluating the preset required performance of the console; A manipulation instruction planning module for, before executing the preset navigation task, obtaining the navigation state information of the simulated transport device and the manipulation state information of the console through the machine vision system; and planning a navigation instruction according to the navigation state information, the manipulation state information, and the preset navigation task to obtain a manipulation instruction instructing the robotic arm to control the console; A manipulation instruction execution module for obtaining mechanical load index data of the robotic arm during the execution of the manipulation instruction, and analyzing the visual display state of the console through the machine vision system to obtain visual load index data; wherein, the mechanical load index data is used to describe the load condition of the operator's arm when the console is actually manipulated, and the visual load index data is used to describe the load condition of the operator's eyes when the console is actually manipulated; A preset performance evaluation module for comprehensively evaluating according to the mechanical load index data and the visual load index data to obtain evaluation data of the console on the preset required performance.

[0014] In a third aspect, an embodiment of the present application provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the method according to any one of the above embodiments.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to cause a computer to execute the method according to any one of the above embodiments.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, including computer instructions for causing a computer to execute the method described in any one of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] Figure 1 It is a step diagram of the auxiliary evaluation method for the control console of the transportation device provided by the embodiment of the present application; Figure 2 It is a schematic structural diagram of the computer-aided evaluation system in the embodiment of the present application; Figure 3 It is a working schematic diagram of the machine vision system in the embodiment of the present application; Figure 4 It is a step diagram of obtaining mechanical load index data in the embodiment of the present application; Figure 5 It is a step diagram of obtaining visual load index data in the embodiment of the present application; Figure 6 It is a step diagram of analyzing the visual display state of the control console in the embodiment of the present application; Figure 7 It is a step diagram of obtaining a manipulation instruction in the embodiment of the present application; Figure 8 It is a step diagram of obtaining a spatial path in the embodiment of the present application; Figure 9 It is a module diagram of the auxiliary evaluation device for the control console of the transportation device provided by the embodiment of the present application; Figure 10 It is a schematic structural diagram of a computer device provided by the embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0020] In common transportation devices, a dedicated console is usually provided to manipulate the transportation device. Transportation devices include vehicles, ships, and aircraft. Exemplarily, for a ship, during the ship's navigation, the driver usually needs to manipulate the ship's console for a long time and continuously maintain a high degree of accuracy and efficiency to cope with complex navigation environments and changing navigation requirements. The ergonomic performance of the ship's console directly affects the driver's manipulation experience. A reasonable ergonomic design can reduce the fatigue level of the driver during manipulation, enabling the driver to maintain the best state. On the contrary, if the ergonomic design of the ship's console is unreasonable, it may cause problems such as rapid physical fatigue or inattention of the driver, increasing the risk of operation errors and thus posing potential safety hazards to the ship's navigation process. Therefore, it is crucial to evaluate the preset requirement performance of the transportation device console before actual use.

[0021] In related technologies, common technical solutions mainly include having multiple testers manipulate the transportation device console and evaluating the preset requirement performance of the transportation device console based on the operation feelings reported by each tester. However, in these technical solutions, subjective human data evaluation is used as the evaluation basis. Due to the influence of the testers' own factors and environmental factors, the feedback results of different testers may have large deviations, resulting in the inability to objectively evaluate the preset requirement performance of the transportation device console and limiting the accuracy of the evaluation results.

[0022] Based on the above problems, the present application discloses a computer-aided evaluation method, device, equipment, and medium for a transportation device console. Among them, this method is applied to a computer-aided evaluation system, which includes a machine vision system, a robotic arm, and a background computer. The machine vision system and the robotic arm are communicatively connected to the background computer, and the background computer is communicatively connected to the console. The method includes: determining a preset navigation task and determining a manipulation instruction; obtaining the mechanical load index data of the robotic arm and the visual load index data of the machine vision system when executing the manipulation instruction, and conducting a comprehensive evaluation to obtain evaluation data.

[0023] The auxiliary evaluation method for the transportation device console disclosed in the present application is applied to a computer-aided evaluation system including a machine vision system, a robotic arm, and a background computer. The background computer is used to control the machine vision system and the robotic arm to execute a preset navigation task and receive the feedback data from the console, the machine vision system, and the robotic arm. In the present application, first, according to the preset navigation task, the machine vision system is used to obtain the navigation state information of the simulated transportation device and the manipulation state information of the console to obtain a manipulation instruction for instructing the robotic arm to perform manipulation; secondly, based on the mechanical load index data feedback by the robotic arm and the visual load index data feedback by the machine vision system, a comprehensive evaluation of the preset requirement performance of the console is conducted to obtain evaluation data.

[0024] Compared with the related art, the present application uses a machine vision system to simulate the eyes of an operator and a robotic arm to simulate the hands of an operator, which not only makes the evaluation conditions controllable and stable, improves the standardization degree of the evaluation process, but also can set quantitative indexes for the preset required performance of the console according to the feedback data of the machine vision system and the robotic arm during the evaluation process for quantitative analysis, thus improving the accuracy of the evaluation results.

[0025] The auxiliary evaluation method for the console of a transportation device provided in this specification can be applied to consoles provided on various transportation devices. The transportation device can include vehicles, ships, aircraft, etc. Among them, the ship can be any one of types such as surface vessels, warships, and submarines navigating underwater. It can be understood that after adaptive modification, the auxiliary evaluation method for the console of a transportation device provided in this specification can also be applied to other mechanical equipment to evaluate these mechanical equipment.

[0026] According to an embodiment of the present application, an embodiment of an auxiliary evaluation method for the console of a transportation device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0027] In this embodiment, an auxiliary evaluation method for the console of a transportation device is provided, which is applied to a computer-aided evaluation system. The computer-aided evaluation system includes a machine vision system, a robotic arm, and a background computer; the console is used to control the simulated transportation device during the evaluation process. The console is within the shooting angle of view of the machine vision system and within the control range of the robotic arm. The machine vision system and the robotic arm are communicatively connected to the background computer, and the background computer is communicatively connected to the console. Refer to Figure 1 As shown, the method includes: S100. Determine a preset navigation task for evaluating the preset required performance of the console.

[0028] S200. Before executing the preset navigation task, obtain the navigation state information of the simulated transportation device and the manipulation state information of the console through the machine vision system; and perform navigation instruction planning according to the navigation state information, the manipulation state information, and the preset navigation task to obtain a manipulation instruction for instructing the robotic arm to manipulate the console.

[0029] S300. Obtain the mechanical load index data of the robotic arm during the execution of the manipulation instruction, and analyze the visual display state of the console through the machine vision system to obtain the visual load index data; wherein, the mechanical load index data is used to describe the load condition of the operator's arm when the console is actually manipulated, and the visual load index data is used to describe the load condition of the operator's eyes when the console is actually manipulated.

[0030] S400. Conduct a comprehensive evaluation based on the mechanical load index data and the visual load index data to obtain the evaluation data of the console on the preset required performance.

[0031] Among them, the preset navigation task can be a task designed in advance according to the actual scenario needs, and is used to display the performance of the console on the preset required performance. The actual execution entity of the preset navigation task is the simulated transportation device in the transportation device simulator, and the control entity of the preset navigation task is the console used to control the simulated transportation device. The transportation device simulator can be a device for simulating the actual navigation state of the transportation device, providing a virtual navigation environment for the simulated transportation device, and then simulating the manipulation process of the operator on the console during the actual navigation process to feedback the feedback data related to the preset required performance of the console.

[0032] The preset required performance can be the ergonomic performance of the console, and the ergonomic performance can be the adaptability between the console and the operator during the actual manipulation of the console by the operator. When the console has excellent ergonomic performance, the console can improve the comfort of the operator during the actual manipulation process, reduce the fatigue caused by the manipulation process, and improve the manipulation performance of the operator. In this application, the ergonomic performance of the console is evaluated according to the mechanical burden and visual burden brought by the console to the operator.

[0033] Refer to Figure 2As shown in the figure, the computer-aided evaluation system includes a machine vision system, a robotic arm, and a background computer. The background computer is communicatively connected to the console, the machine vision system, and the robotic arm respectively. The background computer is used to determine a preset navigation task, control the machine vision system and the robotic arm according to the preset navigation task, and receive the feedback data from the console, the machine vision system, and the robotic arm, so as to evaluate the ergonomic performance of the console. The machine vision system is used to replace the eyes of the testers in the related art, and the machine vision system is used to obtain the visual load index data of the console to describe the load condition of the operator's eyes when the console is actually manipulated. The robotic arm is used to replace the hands of the testers in the related art, and the robotic arm is used to obtain the mechanical load index data of the console during the manipulation process to describe the load condition of the operator's arms when the console is actually manipulated. Using the mechanical load index data and the visual load index data as the data basis, the ergonomic performance of the console is quantitatively analyzed on multiple quantitative indexes, which improves the accuracy of the ergonomic performance evaluation result, and thus can effectively guide the design of the console.

[0034] In some embodiments, the machine vision system can be any one of systems such as a depth camera, multiple GRB cameras, and other multi-camera vision systems, and the robotic arm can be any one of types such as a two-arm robot, an industrial robot, a collaborative robot, and other configured robotic arms, and the number of robotic arms can be increased or decreased according to actual needs.

[0035] Specifically, after determining the preset navigation task, first, the background computer controls the machine vision system to obtain information about the console, and obtains the navigation state information of the simulated transportation device and the manipulation state information of the console in the current situation. It can be understood that this step can simulate the process of the operator obtaining information from the console visually in the actual situation. In some embodiments, the angle of the machine vision system can be adjusted. Exemplarily, according to the known layout of the console, the attitude of the lower pan of the machine vision system is adjusted, so that the angle of the machine vision system can be adjusted to collect information about the console at different angles. In this case, the angle adjustment range of the machine vision system in the preset navigation task can also be used as the visual load index data to participate in the ergonomic performance evaluation of the console.

[0036] Refer to Figure 3As shown, when the machine vision system acquires information, it can collect the left and right images of the console at an angle similar to that of the human eye, and identify the information displayed on the console in the left and right images to obtain the navigation state information and the manipulation state information. Secondly, the background computer uses the navigation state information, the manipulation state information, and the preset navigation task to plan the navigation instructions for the console, and obtains the manipulation instructions for the console based on the preset navigation task, so as to instruct the robotic arm to manipulate the console.

[0037] Further, on the one hand, control the robotic arm to execute the manipulation instructions to complete the manipulation of the console, and collect the feedback data of the robotic arm during the execution of the manipulation instructions as the mechanical load index data. It can be understood that this step can simulate the process of an operator manipulating the console with both hands in an actual situation, and evaluate the mechanical load imposed on the operator by the console based on the mechanical load index data fed back by the robotic arm. On the other hand, analyze the visual display state of the console based on the image of the console collected by the machine vision system to obtain the feedback data of the machine vision system as the visual load index data, so as to evaluate the visual load imposed on the operator by the console.

[0038] Further, based on the obtained mechanical load index data and visual load index data, comprehensively evaluate the ergonomic performance of the console on multiple preset quantitative indicators to obtain the ergonomic evaluation data of the console. In some embodiments, the number and type of the quantitative indicators can be determined according to actual requirements, such as conducting a questionnaire survey on the operator or determining according to relevant literature, etc., so as to improve the comprehensiveness of the evaluation results.

[0039] The auxiliary evaluation method for the transport device console provided in this embodiment is applied to a computer-aided evaluation system including a machine vision system, a robotic arm, and a background computer. The background computer is used to control the machine vision system and the robotic arm to execute the preset navigation task and receive the feedback data of the console, the machine vision system, and the robotic arm. In this application, first, according to the preset navigation task, use the machine vision system to obtain the navigation state information of the simulated transport device and the manipulation state information of the console to obtain the manipulation instructions for instructing the robotic arm to manipulate; secondly, based on the mechanical load index data fed back by the robotic arm and the visual load index data fed back by the machine vision system, comprehensively evaluate the ergonomic performance of the console to obtain the ergonomic evaluation data.

[0040] Compared with the related art, the present application uses a machine vision system to simulate the eyes of an operator and a robotic arm to simulate the hands of an operator, which not only makes the evaluation conditions controllable and stable, improves the standardization of the evaluation process, but also can set quantitative indicators for the ergonomic performance of the console based on the feedback data of the machine vision system and the robotic arm during the evaluation process for quantitative analysis, improving the accuracy of the ergonomic performance evaluation results.

[0041] Referring Figure 4 As shown, as an embodiment of the present application, the transportation device is a ship, the simulated transportation device is a simulated ship, and the console is a ship console; the mechanical load index data is obtained through the following method: S310. Obtain the manipulation torque data of the robotic arm during the execution of the manipulation instruction.

[0042] S312. Count the number of times the robotic arm manipulates the same manipulation component on the ship console during the execution of the manipulation instruction to obtain the component repetition frequency.

[0043] S314. Count the number of times the robotic arm performs the same manipulation action during the execution of the manipulation instruction to obtain the repeated manipulation data.

[0044] S316. Obtain the mechanical load index data based on the manipulation torque data, the component repetition frequency, and the repeated manipulation data.

[0045] In different embodiments, the simulated ship can be a surface vessel or an underwater submarine. It can be understood that the navigation state information of the simulated ship can be fed back by the ship simulator to the ship console and displayed by the ship console.

[0046] Specifically, the manipulation torque data can include one or more of the data such as the joint torque, the end clamping torque, and the load torque of the robotic arm. The above torques respectively describe the torque magnitudes borne by the joints and hands of the operator's hands during the actual manipulation process, as well as the torque magnitudes borne by the hands due to manipulation. It can be understood that the larger the manipulation torque data, the greater the load on the operator's arm caused by manipulating the ship console.

[0047] The component repetition frequency describes the number of times the same manipulation component needs to be manipulated in the preset navigation task. The more the component repetition frequency, the more times the manipulation component needs to be adjusted, or the higher the complexity of the manipulation, which will cause an excessive load on the operator. Similarly, the repeated manipulation data describes the number of times the operator performs the same manipulation action in the preset navigation task. If the same action is repeated for a long time, it will cause chronic fatigue accumulation in the operator, resulting in a decrease in the operator's activity.

[0048] It is understandable that, first, the force conditions of the operator's hands are analyzed based on the control torque data, and second, the fatigue accumulation of the operator's hands is analyzed based on the component repetition frequency and repeated control data, so as to conduct a multi-angle quantitative analysis of the mechanical load on the operator during the preset navigation task and obtain the mechanical load index data of the ship console.

[0049] Referring to Figure 5 As shown, as an embodiment of the present application, the visual load index data is obtained through the following method: S320. Analyze the field of view range of the ship console according to the visual display state to obtain the field of view range data.

[0050] S322. Conduct a visual feature analysis on the visual display state to obtain the visual feature data; among them, the visual feature data includes the data obtained by analyzing the color and brightness of the visual display state.

[0051] S324. Compare the manipulation state information obtained by the machine vision system with the actual manipulation state information of the ship console, and obtain the error recognition data according to the difference between the manipulation state information and the actual manipulation state information.

[0052] S326. Obtain the visual load index data according to the field of view range data, visual feature data and error recognition data.

[0053] Specifically, the field of view range data describes the range that the operator can clearly observe the ship console without moving the line of sight during the actual operation process. The larger the field of view range data, the more compact the design of the ship console, which is suitable for the operator's observation; while the smaller the field of view range data, the more redundant the design of the ship console, causing a visual load on the operator.

[0054] The visual feature data includes the color feature data obtained by analyzing the color of the visual display state, and the brightness feature data obtained by analyzing the brightness of the visual display state. The visual feature data mainly describes the visual fatigue caused by the color setting and brightness setting of the display screen during the actual operation process. Unreasonable color setting and brightness setting will cause the operator's eyes to quickly accumulate fatigue and affect the operator's eyesight.

[0055] Further, the background computer receives the actual operation status information fed back by the ship console through the communication connection with the ship console. The operation status information obtained by the machine vision system is compared with the actual operation status information to determine whether there is information recognition failure caused by unclear display and recognition deviation information different from the actual operation status information in the operation status information obtained by the machine vision system. Error recognition data is obtained based on the recognition failure information and the recognition deviation information, so as to evaluate the information display ability of the display screen of the ship console.

[0056] In some embodiments, the error recognition data may further include the comparison result of the navigation status information. The background computer receives the actual navigation status information fed back by the ship console and compares the navigation status information obtained by the machine vision system with the actual navigation status information to obtain the error recognition data.

[0057] Further, the design of the ship console is analyzed according to the field of view range data, the situation of visual fatigue caused by the ship console to the operator is analyzed according to the visual feature data, and the information display ability of the display screen of the ship console is analyzed according to the error recognition data, so as to conduct a multi-angle quantitative analysis of the visual load suffered by the operator in the preset navigation task and obtain the visual load index data of the ship console.

[0058] Refer to Figure 6 As shown in the figure, as an embodiment of the present application, the ship console includes a display screen and a control component, and the visual display state includes the display screen display state and the control component display state; the visual display state is obtained by the following method: S332. Perform information recognition on the display screen display state to obtain the navigation status information of the simulated ship.

[0059] S334. Perform control component status recognition on the control component display state to obtain the operation status information of the ship console.

[0060] Specifically, the display screen of the ship console usually displays the navigation status information of the ship. In this embodiment, information recognition is performed on the display screen display state collected by the machine vision system to obtain the navigation status information of the simulated ship.

[0061] Further, in the process of passing through Figure 3When the machine vision system shown collects the visual display state of the ship console, it can also analyze the depth of field of the display state of the control components through the left and right images, so as to identify the state of the control components, determine the current state of each control component on the ship console, and obtain the operation state information of the ship console. It can be understood that this step simulates the process of the operator visually confirming the current state of the ship console during the actual operation process, so as to obtain the visual load index data of the ship console on the operator during this process.

[0062] In some embodiments, when identifying the state of the control components, the positions of each control component on the ship console and the relative positions of each control component and the robotic arm can also be calibrated according to the display state of the control components, so as to serve as the data basis for the spatial path planning of the robotic arm.

[0063] Refer to Figure 7 As shown, as an embodiment of the present application, according to the navigation state information, the operation state information, and the preset navigation task, a navigation instruction plan is made to obtain an operation instruction for instructing the robotic arm to control the ship console, including: S210. Make a navigation instruction plan for the simulated ship according to the navigation state information and the navigation target of the simulated ship in the preset navigation task, and obtain a navigation control instruction for controlling the simulated ship.

[0064] S220. Compare the navigation control instruction with the operation state information of the ship console, determine the relative adjustment amount of each control component on the ship console, and obtain an operation instruction.

[0065] Specifically, the preset navigation task includes the navigation target of the simulated ship. After determining the preset navigation task, as well as the navigation state information of the simulated ship and the operation state information of the ship console in the current situation, a navigation instruction plan is made for the simulated ship according to the navigation state information and the preset navigation task, and the navigation control instruction of the simulated ship during the preset navigation task is obtained to determine the navigation mode of the simulated ship. Exemplarily, when the simulated ship is a surface vessel, the navigation control instruction includes the course, speed, and track of the simulated ship, etc.; when the simulated ship is an underwater submarine, the navigation control instruction includes the depth, course, speed, and attitude of the simulated ship, etc.

[0066] Furthermore, compare the target operation state corresponding to the navigation control instruction on the ship console with the current operation state information of the ship console, and obtain the relative adjustment amount of each control component between the target operation state and the operation state information. It can be understood that the relative adjustment amount is the task that needs to control the robotic arm to operate the ship console when executing the preset navigation task. Therefore, an operation instruction for instructing the robotic arm to control the ship console can be obtained according to the relative adjustment amount.

[0067] As an embodiment of the present application, before obtaining the mechanical load index data of the robotic arm during the execution of the manipulation instruction, it includes: S230. Perform spatial path planning on the robotic arm according to the manipulation instruction to obtain a spatial path for controlling the robotic arm.

[0068] Specifically, the object of the spatial path planning is the end of the robotic arm and the gripper. The robotic arm contacts and fixes with the manipulation component through the gripper to achieve the manipulation of the manipulation component. In this embodiment, the type of the robotic arm is a dual-arm robot, including a robot body, a control box, and a gripper. The control box includes a robotic arm control box for controlling the robotic arm in the robot body. A gripper is installed at the end of the robotic arm, which is driven electrically and transmitted through the combination of a connecting rod and a gear.

[0069] Furthermore, during the process of spatial path planning, the spatial path passed by the end of the robotic arm and the gripper is planned according to the total length of the spatial path and whether the spatial path passes through other manipulation components. Among them, the total length of the spatial path describes the length of the spatial path passed by the operator's hand during the actual manipulation process. The shorter the total length of the spatial path, the less the fatigue accumulation of the operator's hand can be reduced, and the manipulation performance of the operator can be guaranteed.

[0070] In some embodiments, during the execution of the preset navigation task, the manipulation instruction can also be modified manually by the tester or automatically by the background computer according to the navigation state information of the simulated ship, so as to interrupt the manipulation instruction at the previous moment and control the robotic arm to execute the new manipulation instruction at the next moment. In this case, the pose of the robotic arm at the start of the new manipulation instruction is not a fixed position, but an unspecified random position. At this time, in order to obtain the spatial path starting from this random position, it is necessary to obtain the pose of the robotic arm through the machine vision system to determine this random position so as to perform spatial path planning.

[0071] Refer to Figure 8 As shown, as an embodiment of the present application, performing spatial path planning on the robotic arm according to the manipulation instruction to obtain a spatial path for controlling the robotic arm includes: S232. Determine a number of path points in the space where the ship console is located according to the outer dimension, component position, and relative adjustment amount of the manipulation component in the manipulation instruction.

[0072] S234. Perform quaternion interpolation between the number of path points to obtain a spatial path passing through the path points.

[0073] Specifically, quaternion interpolation represents the rotation in space through a scalar and three imaginary parts, which can effectively represent the rotation of the manipulation component in space and achieve a smooth rotation transition, enabling the robotic arm to smoothly transition its pose. In this embodiment, the contact point between the robotic arm and the manipulation component is determined based on the external dimensions and position of the manipulation component, serving as the first path point. And the contact point between the robotic arm and the manipulation component after manipulation is determined according to the relative adjustment amount, serving as the second path point. Thus, a spatial path for manipulating the manipulation component is obtained based on the first path point and the second path point. It can be understood that there is a corresponding spatial path for each manipulation component to be manipulated, and the spatial paths between each pair of spatial paths are also connected through the spatial path between components, thereby achieving smooth movement of the robotic arm between different manipulation components.

[0074] Correspondingly, please refer to Figure 9 , an auxiliary evaluation device for a transportation device console provided in an embodiment of the present application is applied to a computer-aided evaluation system. The computer-aided evaluation system includes a machine vision system, a robotic arm, and a background computer; the console is used to control a simulated transportation device during the evaluation process, the console is within the shooting perspective of the machine vision system and within the manipulation range of the robotic arm, the machine vision system and the robotic arm are communicatively connected to the background computer, and the background computer is communicatively connected to the console; the device includes: A preset task determination module 910, configured to determine a preset navigation task for evaluating the preset required performance of the console. A manipulation instruction planning module 920, configured to, before executing the preset navigation task, obtain the navigation state information of the simulated transportation device and the manipulation state information of the console through the machine vision system; and perform navigation instruction planning according to the navigation state information, the manipulation state information, and the preset navigation task to obtain a manipulation instruction for instructing the robotic arm to manipulate the console. A manipulation instruction execution module 930, configured to obtain the mechanical load index data of the robotic arm during the execution of the manipulation instruction, and analyze the visual display state of the console through the machine vision system to obtain visual load index data; wherein, the mechanical load index data is used to describe the load condition of the operator's arm when the console is actually manipulated, and the visual load index data is used to describe the load condition of the operator's eyes when the console is actually manipulated. A preset performance evaluation module 940, configured to perform a comprehensive evaluation according to the mechanical load index data and the visual load index data to obtain evaluation data of the console on the preset required performance.

[0075] In some alternative embodiments, the transportation device is a ship, the simulated transportation device is a simulated ship, and the console is a ship console; the manipulation instruction execution module 930 includes: A manipulation torque acquisition unit, configured to acquire the manipulation torque data of the robotic arm during the execution of the manipulation instruction.

[0076] A repetition frequency statistical unit, which is used to count the number of times the manipulator controls the same manipulation component on the ship console during the execution of the manipulation instruction, so as to obtain the component repetition frequency.

[0077] A repeated action statistical unit, which is used to count the number of times the manipulator performs the same manipulation action during the execution of the manipulation instruction, so as to obtain the repeated manipulation data.

[0078] A mechanical load index data acquisition unit, which is used to obtain the mechanical load index data according to the manipulation torque data, the component repetition frequency and the repeated manipulation data.

[0079] In some alternative embodiments, the manipulation instruction execution module 930 further includes: A field of view range analysis unit, which is used to analyze the field of view range of the ship console according to the visual display state, so as to obtain the field of view range data.

[0080] A visual feature analysis unit, which is used to analyze the visual features of the visual display state, so as to obtain the visual feature data; wherein, the visual feature data includes the data obtained by analyzing the color and brightness of the visual display state.

[0081] A manipulation state comparison unit, which is used to compare the manipulation state information obtained by the machine vision system with the actual manipulation state information of the ship console, and obtain the error identification data according to the difference between the manipulation state information and the actual manipulation state information.

[0082] A visual load index data acquisition unit, which is used to obtain the visual load index data according to the field of view range data, the visual feature data and the error identification data.

[0083] In some alternative embodiments, the ship console includes a display screen and manipulation components, and the visual display state includes the display screen display state and the manipulation component display state; the manipulation instruction execution module 930 further includes: A navigation state information recognition unit, which is used to recognize the information of the display screen display state, so as to obtain the navigation state information of the simulated ship.

[0084] A manipulation state information recognition unit, which is used to recognize the manipulation component state of the manipulation component display state, so as to obtain the manipulation state information of the ship console.

[0085] In some alternative embodiments, the manipulation instruction planning module 920 includes: A navigation instruction planning unit, which is used to plan the navigation instruction for the simulated ship according to the navigation state information and the navigation target of the simulated ship in the preset navigation task, so as to obtain the navigation control instruction for controlling the simulated ship.

[0086] A maneuvering instruction planning unit, configured to compare a navigation control instruction with the maneuvering state information of a ship console, determine the relative adjustment amounts of the various maneuvering components on the ship console, and obtain a maneuvering instruction.

[0087] In some alternative embodiments, the maneuvering instruction planning module 920 further includes: A spatial path planning unit, configured to perform spatial path planning on the robotic arm according to the maneuvering instruction, and obtain a spatial path for controlling the robotic arm.

[0088] In some alternative embodiments, the maneuvering instruction planning module 920 further includes: A path point determination unit, configured to determine a number of path points in the space where the ship console is located according to the external dimensions, component positions, and relative adjustment amounts of the maneuvering components in the maneuvering instruction.

[0089] A path point interpolation unit, configured to perform quaternion interpolation between the number of path points to obtain a spatial path passing through the path points.

[0090] The further function descriptions of the above-mentioned various modules and units are the same as those in the corresponding foregoing embodiments, and will not be elaborated herein.

[0091] The auxiliary evaluation device of the transport device console in this embodiment is presented in the form of functional units. Here, the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0092] Please refer to Figure 10 , Figure 10 FIG. is a schematic structural diagram of a computer device provided by an embodiment of the present application. As shown in the figure, the computer device includes: one or more processors 10, a memory 20, and an interface for connecting each component, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a set of blade servers, or a multi-processor system). Figure 10 In FIG., one processor 10 is taken as an example.

[0093] The processor 10 may be a central processing unit, a network processor, or a combination thereof. Among them, the processor 10 may further include a hardware chip. The above-mentioned hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above-mentioned programmable logic device may be a complex programmable logic device, a field-programmable gate array, a generic array logic, or any combination thereof.

[0094] Among them, the memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiments.

[0095] The memory 20 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the computer device. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 may optionally include a memory remotely provided with respect to the processor 10, and these remote memories may be connected to the computer device through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0096] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk, or a solid-state drive; the memory 20 may also include a combination of the above types of memories.

[0097] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0098] The embodiments of the present application further provide a computer-readable storage medium. The method according to the embodiments of the present application may be implemented in hardware, firmware, or may be implemented as computer code recorded on a storage medium, or may be implemented as computer code originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and stored in a local storage medium, so that the method described herein may be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium may also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0099] An embodiment of the present application provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of any embodiment of the present application.

[0100] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations fall within the scope defined by the appended claims.

[0101] The systems, devices, modules or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0102] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in one or more software and / or hardware.

[0103] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate for implementing in the process Figure 1 one process or multiple processes and / or blocks Figure 1a device for the functions specified in one or more boxes

[0105] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device, and the instruction device implements the process Figure 1 one process or multiple processes and / or boxes Figure 1 the functions specified in one box or multiple boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one process or multiple processes and / or boxes Figure 1 the functions specified in one box or multiple boxes.

[0107] It should also be noted that the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such a process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, commodity or device including the said element.

[0108] Each embodiment in this specification is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and the key points of each embodiment are the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment.

[0109] The above description is only for the embodiments of this application and is not intended to limit this application. For those skilled in the art, this application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

[0110] Although the embodiments of this application are described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method for assisting evaluation of a transport device console, characterized in that: The method is applied to a computer-aided evaluation system, the computer-aided evaluation system includes a machine vision system, a mechanical arm and a background computer; the control console is used to control the simulated transportation device during the evaluation process, the control console is located within the shooting angle of view of the machine vision system and within the control range of the mechanical arm, the machine vision system and the mechanical arm are connected to the background computer in communication, and the background computer is connected to the control console in communication; the method includes: determining a predetermined navigation task for evaluating a predetermined required performance of the control station; Before executing the preset navigation task, the navigation state information of the simulated transportation device and the manipulation state information of the control panel are obtained through the machine vision system; and navigation instruction planning is performed according to the navigation state information, the manipulation state information and the preset navigation task to obtain the manipulation instruction instructing the mechanical arm to manipulate the control panel; Acquiring mechanical load index data of the mechanical arm in the process of executing the manipulation instruction, and analyzing the visual display state of the control panel through the machine vision system to obtain visual load index data; wherein the mechanical load index data is used to describe the load condition of the operator's arm when the control panel is actually manipulated, and the visual load index data is used to describe the load condition of the operator's eyes when the control panel is actually manipulated; A comprehensive evaluation is performed based on the mechanical load index data and the visual load index data to obtain evaluation data of the control console in terms of the preset required performance.

2. The method according to claim 1, characterized in that The transport device is a ship, the simulated transport device is a simulated ship, and the control panel is a ship control panel; The mechanical load index data is obtained by: Acquiring control torque data of the robot arm during the execution of the control instruction; Counting the number of times the mechanical arm manipulates the same manipulation component on the ship control console during the execution of the manipulation instruction, to obtain a component repetition frequency; Counting the number of times the robot arm performs the same manipulation action in the process of executing the manipulation instruction to obtain repeated manipulation data; The mechanical load index data is obtained according to the control torque data, the component repetition frequency and the repeated control data.

3. The method according to claim 2, characterized in that The visual load index data is obtained by: Performing a visual range analysis on the ship control console according to the visual display state to obtain visual range data; Performing visual feature analysis on the visual display state to obtain visual feature data; wherein the visual feature data includes data obtained by performing color analysis and brightness analysis on the visual display state; comparing the control state information acquired by the machine vision system with the actual control state information of the ship control console, and obtaining error recognition data according to the difference between the control state information and the actual control state information; The visual load index data is obtained according to the visual field data, the visual feature data and the error recognition data.

4. The method according to claim 2, characterized in that: The ship control console includes a display screen and a control component, and the visual display state includes a display screen display state and a control component display state; The visual display state is obtained by: Performing information recognition on the display status of the display screen to obtain navigation status information of the simulated ship; The control component status is identified on the display status of the control component to obtain the control status information of the ship control panel.

5. The method according to claim 2, characterized in that: The performing of navigation instruction planning according to the navigation state information, the manipulation state information and the preset navigation task to obtain the manipulation instruction instructing the mechanical arm to manipulate the ship control console includes: Planning navigation instructions for the simulated ship according to the navigation status information and the navigation target of the simulated ship in the preset navigation task, and obtaining navigation control instructions for controlling the simulated ship; The navigation control instruction is compared with the control state information of the ship control panel, the relative adjustment amount of each control component on the ship control panel is determined, and the control instruction is obtained.

6. The method according to claim 2, characterized in that Before obtaining the mechanical load index data of the mechanical arm during the execution of the manipulation instruction, the method includes: The spatial path of the robot arm is planned according to the manipulation instruction to obtain a spatial path for controlling the robot arm.

7. The method according to claim 6, characterized in that The performing spatial path planning on the robot arm according to the manipulation instruction to obtain a spatial path for controlling the robot arm includes: Determine a plurality of path points in the space where the ship control console is located according to the outer dimensions, component positions and relative adjustment amounts of the control components in the control instructions; Quaternion interpolation is performed between the plurality of path points to obtain a spatial path passing through the path points.

8. An auxiliary evaluation device for a transport device console, characterized in that: Applied to a computer-aided evaluation system, the computer-aided evaluation system includes a machine vision system, a mechanical arm and a background computer; the console is used to control the simulated transportation device during the evaluation process, the console is located within the shooting angle of the machine vision system and within the control range of the mechanical arm, the machine vision system and the mechanical arm are connected to the background computer in communication, and the background computer is connected to the console in communication; the device includes: A preset task determination module, used to determine a preset navigation task for evaluating the preset required performance of the control console; A manipulation instruction planning module is used to obtain the navigation state information of the simulated transportation device and the manipulation state information of the control console through the machine vision system before executing the preset navigation task; and perform navigation instruction planning according to the navigation state information, the manipulation state information and the preset navigation task to obtain the manipulation instruction instructing the mechanical arm to manipulate the control console; A manipulation instruction execution module, used to obtain mechanical load index data of the mechanical arm in the process of executing the manipulation instruction, and to analyze the visual display state of the control panel through the machine vision system to obtain visual load index data; wherein the mechanical load index data is used to describe the load condition of the operator's arm when the control panel is actually manipulated, and the visual load index data is used to describe the load condition of the operator's eyes when the control panel is actually manipulated; The preset performance evaluation module is used to perform a comprehensive evaluation based on the mechanical load index data and the visual load index data to obtain evaluation data of the control console in terms of the preset required performance.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images