Auxiliary evaluation method, device, equipment and medium for a transport device control panel

Abstract

The present application relates to the field of computer-aided technology, and discloses an auxiliary evaluation method, device, equipment and medium for a transport device console, which is applied to a computer-aided evaluation system. The computer-aided evaluation system includes a machine vision system, a robotic arm and a background computer. The machine vision system and the robotic arm are communicatively connected to the background computer, and the background computer is communicatively connected to the console. The method includes: determining a preset navigation task and determining a manipulation instruction; obtaining the mechanical load index data of the robotic arm and the visual load index data of the machine vision system when executing the manipulation instruction, and performing a comprehensive evaluation to obtain evaluation data on the preset required performance. The beneficial effect of the present application is that the robotic arm is used to simulate the hands of the operator, and the machine vision system is used to simulate the eyes of the operator, and the preset required performance of the console is quantitatively evaluated based on the feedback data of the robotic arm and the machine vision system, thereby improving the accuracy of the evaluation results.

Description

Technical Field

[0001] The present application relates to the field of computer-aided technology, and in particular to an auxiliary evaluation method, device, equipment, and medium for a transport device console. Background Art

[0002] Common transportation devices are usually equipped with a special control panel to operate the transportation device. Transportation devices include vehicles, ships and aircraft. For example, for ships, during the navigation process, the driver usually needs to operate the ship's control panel for a long time and continue to maintain a high degree of accuracy and efficiency to cope with complex navigation environments and changing navigation requirements. The ergonomic performance of the ship's control panel directly affects the driver's control experience. Reasonable ergonomic design can reduce the driver's fatigue during the operation process, allowing the driver to maintain the best condition. On the contrary, if the ergonomic design of the ship's control panel is unreasonable, it may cause the driver to quickly become physically tired or lose concentration, increase the risk of operational errors, and thus bring safety hazards to the ship's navigation process. Therefore, it is very important to evaluate the ergonomic performance of the transportation device control panel before it is actually put into use.

[0003] In the related art, subjective data is usually used to evaluate the preset required performance of the transportation device control panel, but the objectivity and accuracy of the related art still need to be improved. Summary of the Invention

[0004] The present application provides an auxiliary evaluation method, device, equipment and medium for a transport device console, which utilizes a robotic arm to simulate the operator's hands and a machine vision system to simulate the operator's eyes, and implements a quantitative evaluation of the preset required performance of the transport device console based on feedback data from the robotic arm and the machine vision system, thereby improving the accuracy of the evaluation results.

[0005] In order to achieve the above objectives, the main technical solutions adopted in this application include:

[0006] In a first aspect, embodiments of the present application provide a method for assisting in the evaluation of a transport device console, the method being applied to a computer-assisted evaluation system, the computer-assisted evaluation system comprising a machine vision system, a robotic arm, and a background computer; the console being used to control a simulated transport device during the evaluation process, the console being located within the viewing angle of the machine vision system and within the control range of the robotic arm; the machine vision system and the robotic arm being communicatively connected to the background computer, and the background computer being communicatively connected to the console; the method comprising:

[0007] determining a predetermined navigation task for evaluating a predetermined required performance of the control panel;

[0008] Before executing the preset navigation task, obtaining navigation status information of the simulated transport device and manipulation status information of the control panel through the machine vision system; and performing navigation instruction planning based on the navigation status information, the manipulation status information, and the preset navigation task to obtain a manipulation instruction instructing the robotic arm to manipulate the control panel;

[0009] Acquiring mechanical load index data of the robotic arm during the execution of the manipulation instruction, and analyzing the visual display state of the console through the machine vision system to obtain visual load index data; wherein the mechanical load index data is used to describe the load on the operator's arm when the console is actually manipulated, and the visual load index data is used to describe the load on the operator's eyes when the console is actually manipulated;

[0010] A comprehensive evaluation is performed based on the mechanical load index data and the visual load index data to obtain evaluation data of the control panel in terms of the preset required performance.

[0011] The present invention proposes an assisted evaluation method for a transport device console, which is applied to a computer-assisted evaluation system comprising a machine vision system, a robotic arm, and a backend computer. The backend computer is used to control the machine vision system and the robotic arm to execute a preset navigation task and receive feedback data from the console, the machine vision system, and the robotic arm. First, according to the preset navigation task, the machine vision system is used to obtain navigation status information of the simulated transport device and control status information of the console to obtain control instructions for the robotic arm to perform operations. Second, based on the mechanical load index data fed back by the robotic arm and the visual load index data fed back by the machine vision system, a comprehensive evaluation of the preset required performance of the console is performed to obtain evaluation data. Compared to related technologies, the present invention utilizes the machine vision system to simulate the operator's eyes and the robotic arm to simulate the operator's hands. This not only makes the evaluation conditions controllable and stable, improving the standardization of the evaluation process, but also enables quantitative analysis of the preset required performance of the console based on the feedback data from the machine vision system and the robotic arm during the evaluation process, thereby improving the accuracy of the evaluation results.

[0012] Optionally, the transport device is a ship, the simulated transport device is a simulated ship, and the control console is a ship control console; the mechanical load index data is obtained by:

[0013] Acquiring manipulation torque data of the robotic arm during execution of the manipulation instruction;

[0014] Counting the number of times the manipulator arm manipulates the same manipulation component on the ship control console during the execution of the manipulation instruction to obtain a component repetition frequency;

[0015] Counting the number of times the robot arm performs the same manipulation action during the execution of the manipulation instruction to obtain repeated manipulation data;

[0016] The mechanical load index data is obtained according to the control torque data, the component repetition frequency and the repeated control data.

[0017] Optionally, the visual load indicator data is obtained by:

[0018] Performing a visual range analysis on the ship control console according to the visual display state to obtain visual range data;

[0019] Performing visual feature analysis on the visual display state to obtain visual feature data; wherein the visual feature data includes data obtained by performing color analysis and brightness analysis on the visual display state;

[0020] comparing the manipulation state information acquired by the machine vision system with actual manipulation state information of the ship control console, and obtaining error recognition data according to a difference between the manipulation state information and the actual manipulation state information;

[0021] The visual load index data is obtained according to the visual field range data, the visual feature data and the error recognition data.

[0022] Optionally, the ship control console includes a display screen and a control component, and the visual display state includes a display screen display state and a control component display state; the visual display state is obtained by:

[0023] Performing information recognition on the display screen display status to obtain navigation status information of the simulated ship;

[0024] The control component display status is identified to obtain the control status information of the ship control panel.

[0025] Optionally, the performing navigation instruction planning according to the navigation state information, the manipulation state information and the preset navigation task to obtain a manipulation instruction instructing the manipulator arm to manipulate the ship control console includes:

[0026] performing navigation instruction planning for the simulated ship according to the navigation state information and the navigation target of the simulated ship in the preset navigation mission, and obtaining navigation control instructions for controlling the simulated ship;

[0027] The navigation control instruction is compared with the control state information of the ship control panel, and the relative adjustment amount of each control component on the ship control panel is determined to obtain the control instruction.

[0028] Optionally, before obtaining the mechanical load index data of the robotic arm during the execution of the manipulation instruction, the method further includes:

[0029] A spatial path planning is performed on the robotic arm according to the manipulation instruction to obtain a spatial path for controlling the robotic arm.

[0030] Optionally, performing spatial path planning on the robotic arm according to the manipulation instruction to obtain a spatial path for controlling the robotic arm includes:

[0031] Determining a plurality of path points in the space where the ship control panel is located according to the outer dimensions, component positions, and relative adjustment amounts of the control components in the control instructions;

[0032] Quaternion interpolation is performed between the plurality of path points to obtain a spatial path passing through the path points.

[0033] In a second aspect, embodiments of the present application provide an auxiliary evaluation device for a transport device console, which is applied to a computer-assisted evaluation system. The computer-assisted evaluation system includes a machine vision system, a robotic arm, and a background computer. The console is used to control a simulated transport device during the evaluation process. The console is located within the viewing angle of the machine vision system and within the control range of the robotic arm. The machine vision system and the robotic arm are communicatively connected to the background computer, and the background computer is communicatively connected to the console. The device includes:

[0034] a preset task determination module, configured to determine a preset navigation task for evaluating the preset required performance of the control console;

[0035] a manipulation instruction planning module, configured to obtain, through the machine vision system, navigation state information of the simulated transport device and manipulation state information of the control console before executing the preset navigation task; and perform navigation instruction planning based on the navigation state information, the manipulation state information, and the preset navigation task to obtain a manipulation instruction for instructing the robotic arm to manipulate the control console;

[0036] a manipulation instruction execution module, configured to obtain mechanical load index data of the manipulator arm during the execution of the manipulation instruction, and to analyze the visual display state of the control panel through the machine vision system to obtain visual load index data; wherein the mechanical load index data is used to describe the load on the operator's arm when the control panel is actually being manipulated, and the visual load index data is used to describe the load on the operator's eyes when the control panel is actually being manipulated;

[0037] The preset performance evaluation module is used to perform a comprehensive evaluation based on the mechanical load index data and the visual load index data to obtain evaluation data of the control console in terms of the preset required performance.

[0038] In a third aspect, an embodiment of the present application provides a computer device comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the method described in any one of the above embodiments by executing the computer instructions.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute any one of the methods in the above embodiments.

[0040] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute any one of the methods described in the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 A diagram showing the steps of the auxiliary evaluation method for a transport device console provided in an embodiment of the present application;

[0043] Figure 2 This is a schematic diagram of the structure of the computer-aided evaluation system in an embodiment of the present application;

[0044] Figure 3 This is a working diagram of the machine vision system in an embodiment of the present application;

[0045] Figure 4 A diagram showing the steps for obtaining mechanical load index data in an embodiment of the present application;

[0046] Figure 5 A diagram showing the steps for obtaining visual load index data in an embodiment of the present application;

[0047] Figure 6 A diagram showing the steps of analyzing the visual display state of the console in an embodiment of the present application;

[0048] Figure 7 A diagram showing the steps for obtaining a manipulation instruction in an embodiment of the present application;

[0049] Figure 8 A diagram showing the steps for obtaining a spatial path in an embodiment of the present application;

[0050] Figure 9 A module diagram of an auxiliary evaluation device for a transport device console provided in an embodiment of the present application;

[0051] Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] To make the purpose, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.

[0053] Common transportation devices are usually equipped with a special control panel to operate the transportation device. Transportation devices include vehicles, ships and aircraft. For example, for ships, during the navigation process, the driver usually needs to operate the ship's control panel for a long time and continue to maintain a high degree of accuracy and efficiency to cope with complex navigation environments and changing navigation requirements. The ergonomic performance of the ship's control panel directly affects the driver's control experience. Reasonable ergonomic design can reduce the driver's fatigue during the operation process, allowing the driver to maintain the best condition. On the contrary, if the ergonomic design of the ship's control panel is unreasonable, it may cause the driver to quickly become physically tired or lose concentration, increase the risk of operational errors, and thus bring safety hazards to the ship's navigation process. Therefore, it is crucial to conduct a preset performance evaluation of the transportation device control panel before it is actually put into use.

[0054] In related technologies, a common approach involves having multiple testers operate a transport device console and evaluating the console's performance against its pre-set requirements based on each tester's feedback. However, these approaches rely on subjective human data evaluation as the basis for evaluation. Due to the influence of testers' own and environmental factors, feedback from different testers can vary significantly, making it impossible to objectively evaluate the transport device console's performance, limiting the accuracy of the evaluation results.

[0055] To address the above-mentioned issues, this application discloses a computer-assisted evaluation method, apparatus, device, and medium for a transport device control panel. The method is applied to a computer-assisted evaluation system comprising a machine vision system, a robotic arm, and a backend computer. The machine vision system and the robotic arm are in communication with the backend computer, which is in communication with the control panel. The method includes: determining a preset navigation mission and determining a maneuvering instruction; obtaining mechanical load index data of the robotic arm and visual load index data of the machine vision system when executing the maneuvering instruction; and performing a comprehensive evaluation to obtain evaluation data.

[0056] The auxiliary evaluation method for a transport device control panel disclosed in this application is applied to a computer-assisted evaluation system including a machine vision system, a robotic arm, and a background computer. The background computer is used to control the machine vision system and the robotic arm to perform preset navigation tasks and receive feedback data from the control panel, the machine vision system, and the robotic arm. In this application, first, based on the preset navigation task, the machine vision system is used to obtain navigation status information of the simulated transport device and control status information of the control panel to obtain control instructions instructing the robotic arm to perform control. Secondly, based on the mechanical load index data fed back by the robotic arm and the visual load index data fed back by the machine vision system, the preset required performance of the control panel is comprehensively evaluated to obtain evaluation data.

[0057] Compared with related technologies, this application uses a machine vision system to simulate the operator's eyes and a robotic arm to simulate the operator's hands. This not only makes the evaluation conditions controllable and stable, and improves the standardization of the evaluation process, but also can set quantitative indicators for the preset required performance of the console based on the feedback data of the machine vision system and the robotic arm during the evaluation process for quantitative analysis, thereby improving the accuracy of the evaluation results.

[0058] The assisted evaluation method for transport device consoles provided in this specification can be applied to consoles installed on a variety of transport devices, including vehicles, ships, and aircraft. Ships can be any type of surface vessel, warship, or underwater submarine. It is understood that, after adaptive modification, the assisted evaluation method for transport device consoles provided in this specification can also be applied to other mechanical equipment to evaluate such equipment.

[0059] According to an embodiment of the present application, an embodiment of an auxiliary evaluation method for a transport device control panel is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0060] In this embodiment, a method for assisting the evaluation of a transport device console is provided, which is applied to a computer-assisted evaluation system. The computer-assisted evaluation system includes a machine vision system, a robotic arm, and a background computer. The console is used to control the simulated transport device during the evaluation process. The console is located within the shooting angle of the machine vision system and within the control range of the robotic arm. The machine vision system and the robotic arm are in communication with the background computer, and the background computer is in communication with the console. Figure 1 As shown, the method includes:

[0061] S100. Determine a preset navigation task for evaluating the preset required performance of the control panel.

[0062] S200. Before executing the preset navigation task, obtain the navigation status information of the simulated transportation device and the control status information of the control panel through the machine vision system; and plan the navigation instructions based on the navigation status information, the control status information and the preset navigation task to obtain the control instructions for instructing the robotic arm to operate the control panel.

[0063] S300. Obtain the mechanical load index data of the robotic arm during the execution of the manipulation instruction, and analyze the visual display status of the control panel through the machine vision system to obtain the visual load index data; wherein, the mechanical load index data is used to describe the load condition of the operator's arm when the control panel is actually manipulated, and the visual load index data is used to describe the load condition of the operator's eyes when the control panel is actually manipulated.

[0064] S400. Perform a comprehensive evaluation based on the mechanical load index data and the visual load index data to obtain evaluation data of the control panel in terms of the preset required performance.

[0065] Preset navigation missions can be pre-designed based on actual scenario requirements to demonstrate the console's performance at preset performance requirements. The simulated transport within the transport simulator executes these missions, and the console used to control the simulated transport is the controller. The transport simulator can be a device used to simulate the actual navigation of a transport, providing a virtual navigation environment for the simulated transport. This simulation then simulates the operator's control of the console during actual navigation, providing feedback data related to the console's preset performance requirements.

[0066] The preset required performance can be the ergonomic performance of the control panel. Ergonomic performance can be the compatibility between the control panel and the operator during actual operation of the control panel. When the control panel has excellent ergonomic performance, it can improve the operator's comfort during actual operation, reduce fatigue caused by the operation process, and improve the operator's control performance. In this application, the ergonomic performance of the control panel is evaluated based on the mechanical and visual burden imposed on the operator by the control panel.

[0067] Reference Figure 2 As shown, the computer-assisted evaluation system includes a machine vision system, a robotic arm, and a backend computer. The backend computer is connected to the control console, the machine vision system, and the robotic arm, respectively. The backend computer is used to determine a preset navigation task, control the machine vision system and the robotic arm according to the preset navigation task, and receive feedback data from the control console, the machine vision system, and the robotic arm to evaluate the ergonomic performance of the control console. The machine vision system replaces the tester's eyes in related technologies and uses it to obtain visual load index data of the control console to describe the load on the operator's eyes when the control console is actually operated. The robotic arm replaces the tester's hands in related technologies and uses it to obtain mechanical load index data of the control console during operation to describe the load on the operator's arms when the control console is actually operated. Using the mechanical load index data and the visual load index data as the data foundation, the ergonomic performance of the control console can be quantitatively analyzed based on multiple quantitative indicators, improving the accuracy of the ergonomic performance evaluation results and effectively guiding the design of the control console.

[0068] In some embodiments, the machine vision system can be any one of a depth camera, multiple GRB cameras, and other multi-eye vision systems, and the robotic arm can be any one of a dual-arm robot, an industrial robot, a collaborative robot, and other configuration robotic arms, and the number of robotic arms can be increased or decreased according to actual needs.

[0069] Specifically, after determining the preset navigation task, the machine vision system is first controlled by the background computer to obtain information from the control console, and obtain the navigation status information of the simulated transportation device and the control status information of the control console under the current situation. It can be understood that this step can simulate the process of the operator obtaining information from the control console through visual observation under actual circumstances. In some embodiments, the angle of the machine vision system is adjustable. For example, according to the known layout of the control console, the posture of the pan-tilt platform under the machine vision system is adjusted, so that the angle of the machine vision system can be adjusted to collect information from the control console at different angles. In this case, the angle adjustment range of the machine vision system in the preset navigation task can also be used as visual load indicator data to participate in the ergonomic performance evaluation of the control console.

[0070] Reference Figure 3 As shown, when acquiring information, the machine vision system captures the left and right images of the console at angles similar to those of the human eye. By identifying the information displayed on the console in these images, the system obtains navigation and manipulation status information. Next, a backend computer plans navigation instructions for the console based on this navigation and manipulation information and the pre-set navigation mission. This command is then used to instruct the robotic arm to operate the console.

[0071] Furthermore, on one hand, the robotic arm is controlled to execute manipulation instructions to complete the operation of the control panel, and feedback data from the robotic arm during the execution of the manipulation instructions is collected as mechanical load index data. It is understood that this step can simulate the process of an operator manipulating the control panel with both hands under actual circumstances, and the mechanical load imposed on the operator by the control panel is evaluated based on the mechanical load index data fed back by the robotic arm. On the other hand, based on the image of the control panel captured by the machine vision system, the visual display state of the control panel is analyzed to obtain feedback data from the machine vision system as visual load index data to evaluate the visual load imposed on the operator by the control panel.

[0072] Furthermore, based on the obtained mechanical load index data and visual load index data, a comprehensive evaluation of the ergonomic performance of the control panel is performed based on multiple preset quantitative indicators to obtain ergonomic evaluation data for the control panel. In some embodiments, the number and type of quantitative indicators can be determined based on actual requirements, such as by conducting a questionnaire survey of operators or by citing relevant literature, thereby improving the comprehensiveness of the evaluation results.

[0073] The auxiliary evaluation method for a transport device control panel provided in this embodiment is applied to a computer-assisted evaluation system comprising a machine vision system, a robotic arm, and a background computer. The background computer is used to control the machine vision system and the robotic arm to execute a preset navigation task and receive feedback data from the control panel, the machine vision system, and the robotic arm. In this application, first, based on the preset navigation task, the machine vision system is used to obtain navigation status information of the simulated transport device and control status information of the control panel to obtain control instructions for instructing the robotic arm to perform control. Second, based on the mechanical load index data fed back by the robotic arm and the visual load index data fed back by the machine vision system, the ergonomic performance of the control panel is comprehensively evaluated to obtain ergonomic evaluation data.

[0074] Compared with related technologies, this application uses a machine vision system to simulate the operator's eyes and a robotic arm to simulate the operator's hands. This not only makes the evaluation conditions controllable and stable, and improves the standardization of the evaluation process, but also can set quantitative indicators for the ergonomic performance of the console based on the feedback data of the machine vision system and the robotic arm during the evaluation process for quantitative analysis, thereby improving the accuracy of the ergonomic performance evaluation results.

[0075] Reference Figure 4 As shown, as an embodiment of the present application, the transport device is a ship, the simulated transport device is a simulated ship, and the control console is a ship control console; the mechanical load index data is obtained by the following method:

[0076] S310. Obtain the manipulation torque data of the robotic arm during the execution of the manipulation instruction.

[0077] S312. Count the number of times the manipulator arm manipulates the same manipulation component on the ship's control panel during the execution of the manipulation instruction to obtain the component repetition frequency.

[0078] S314. Count the number of times the robot arm performs the same manipulation action during the execution of the manipulation instruction to obtain repeated manipulation data.

[0079] S316. Obtain mechanical load index data based on the control torque data, component repetition frequency and repeated control data.

[0080] In different embodiments, the simulated ship can be a surface ship or an underwater submarine. It is understandable that the navigation status information of the simulated ship can be fed back to the ship control console by the ship simulator and displayed by the ship control console.

[0081] Specifically, the control torque data may include one or more of the manipulator's joint torque, end-of-line clamping torque, and load torque. These torques describe the torques experienced by the operator's hand joints and hands during actual control, as well as the torques experienced by both hands due to the manipulation. It is understood that a greater control torque indicates a greater load on the operator's arms caused by operating the vessel's control console.

[0082] Component repetition frequency describes the number of times the same control component needs to be manipulated during a given mission. A higher component repetition frequency indicates that the control component requires frequent adjustment or that the control is too complex, placing excessive strain on the operator. Similarly, repetition data describes the number of times the operator performs the same maneuver during a given mission. Prolonged repetition of the same maneuver can lead to chronic fatigue and decreased operator performance.

[0083] It can be understood that first, the force conditions of the operator's hands are analyzed based on the control torque data, and then the fatigue accumulation of the operator's hands is analyzed based on the component repetition frequency and repeated control data, so as to conduct a multi-angle quantitative analysis of the mechanical loads on the operator in the preset navigation tasks and obtain the mechanical load index data of the ship's control panel.

[0084] Reference Figure 5 As shown, as an embodiment of the present application, visual load index data is obtained in the following manner:

[0085] S320. Analyze the visual range of the ship's control panel based on the visual display status to obtain visual range data.

[0086] S322. Perform visual feature analysis on the visual display state to obtain visual feature data; wherein the visual feature data includes data obtained by performing color analysis and brightness analysis on the visual display state.

[0087] S324. Compare the control state information obtained by the machine vision system with the actual control state information of the ship control panel, and obtain error recognition data based on the difference between the control state information and the actual control state information.

[0088] S326. Obtain visual load index data based on the visual field range data, visual feature data and error recognition data.

[0089] Specifically, the field of view data describes the range of the ship's control panel that the operator can clearly observe without shifting their sight during actual operation. A larger field of view indicates a more compact control panel design, making it easier for the operator to observe. A smaller field of view indicates redundant design, creating a visual overload for the operator.

[0090] Visual feature data includes color feature data obtained through color analysis of the visual display state, and brightness feature data obtained through brightness analysis of the visual display state. This visual feature data primarily describes the extent to which the color and brightness settings of the display screen cause visual fatigue to the operator during actual operation. Improper color and brightness settings can rapidly fatigue the operator's eyes and affect their vision.

[0091] Furthermore, the backend computer receives actual control status information from the ship's control console via a communication connection. The computer compares the control status information obtained by the machine vision system with the actual control status information to determine whether the control status information obtained by the machine vision system contains any recognition failures due to unclear display, or any recognition deviations indicating discrepancies from the actual control status information. Based on these recognition failure and deviation information, the computer generates error recognition data, which is then used to evaluate the information display capabilities of the ship's control console display.

[0092] In some embodiments, the error identification data may also include the comparison results of the navigation status information. The actual navigation status information fed back by the ship's control panel is received through the background computer, and the navigation status information obtained by the machine vision system is compared with the actual navigation status information to obtain the error identification data.

[0093] Furthermore, the design of the ship's control panel is analyzed based on the field of view data, the visual fatigue caused by the ship's control panel to the operator is analyzed based on the visual feature data, and the information display capability of the display screen of the ship's control panel is analyzed based on the error recognition data. This allows for a multi-angle quantitative analysis of the visual load experienced by the operator during the preset navigation mission, and yields visual load index data for the ship's control panel.

[0094] Reference Figure 6 As shown, as an embodiment of the present application, the ship control console includes a display screen and a control component, and the visual display state includes the display screen display state and the control component display state; the visual display state is obtained by the following method:

[0095] S332. Identify the information displayed on the display screen to obtain the navigation status information of the simulated ship.

[0096] S334. Perform control component status identification on the display status of the control component to obtain the control status information of the ship control panel.

[0097] Specifically, the display screen of the ship's control panel usually displays the navigation status information of the ship. In this embodiment, the navigation status information of the simulated ship is obtained by identifying the information displayed on the display screen collected by the machine vision system.

[0098] Furthermore, by Figure 3 When capturing the visual display status of the ship's control panel, the machine vision system can also analyze the depth of field of the control component display status using the left and right images. This allows for control component status recognition, determining the current status of each control component on the ship's control panel, and obtaining control status information for the ship's control panel. This step simulates the actual process of an operator visually confirming the current status of the ship's control panel during operation, thereby obtaining visual load indicators for the operator during this process.

[0099] In some embodiments, when identifying the status of control components, the position of each control component on the ship control panel and the relative position of each control component and the robotic arm can be calibrated according to the display status of the control component to serve as the data basis for spatial path planning of the robotic arm.

[0100] Reference Figure 7 As shown, as an embodiment of the present application, navigation instruction planning is performed according to navigation status information, manipulation status information and preset navigation tasks to obtain a manipulation instruction for instructing the manipulator arm to manipulate the ship's control panel, including:

[0101] S210. Plan navigation instructions for the simulated ship according to the navigation status information and the navigation target of the simulated ship in the preset navigation mission, and obtain navigation control instructions for controlling the simulated ship.

[0102] S220. Compare the navigation control instructions with the control status information of the ship control panel, determine the relative adjustment amount of each control component on the ship control panel, and obtain the control instructions.

[0103] Specifically, the preset navigation mission includes the simulated ship's navigation target. After determining the preset navigation mission, as well as the simulated ship's current navigation status information and the ship's control console's control status information, navigation instructions are planned for the simulated ship based on the navigation status information and the preset navigation mission. Sailing control instructions for the simulated ship during the preset navigation mission are obtained to determine the simulated ship's navigation mode. For example, when the simulated ship is a surface vessel, the navigation control instructions include the simulated ship's heading, speed, and track; when the simulated ship is an underwater submarine, the navigation control instructions include the simulated ship's depth, heading, speed, and attitude.

[0104] Furthermore, by comparing the target control state corresponding to the navigation control command on the ship's control console with the current control state information of the ship's control console, a relative adjustment amount for each control component between the target control state and the control state information is obtained. It will be understood that the relative adjustment amount refers to the task required to control the manipulator arm to manipulate the ship's control console when executing the preset navigation task. Therefore, based on the relative adjustment amount, a control instruction instructing the manipulator arm to manipulate the ship's control console can be obtained.

[0105] As an embodiment of the present application, before obtaining the mechanical load index data of the robot arm during the execution of the manipulation instruction, the method includes:

[0106] S230. Perform spatial path planning for the robotic arm according to the manipulation instruction to obtain a spatial path for controlling the robotic arm.

[0107] Specifically, the spatial path planning targets the end of the robotic arm and its gripper. The gripper contacts and secures the robotic arm to the manipulator, enabling manipulation of the manipulator. In this embodiment, the robotic arm is a dual-arm robot, comprising a robot body, a control box, and grippers. The control box includes a robotic arm control box, which controls the robotic arm within the robot body. The gripper is electrically driven, with transmission via a connecting rod and gears.

[0108] Furthermore, during spatial path planning, the spatial path traversed by the manipulator's end arm and gripper is planned based on the total length of the spatial path and whether the spatial path passes through other control components. The total length of the spatial path describes the length of the spatial path traversed by the operator's hand during actual manipulation. A shorter total length of the spatial path reduces hand fatigue and ensures optimal control performance.

[0109] In some embodiments, during the execution of a preset navigation task, the tester can manually or the background computer can automatically modify the maneuvering instructions based on the navigation status information of the simulated ship, thereby interrupting the maneuvering instructions at the previous moment to control the robotic arm to execute the new maneuvering instructions at the next moment. In this case, the position of the robotic arm at the beginning of the new maneuvering instruction is not a fixed position, but an unspecified random position. At this time, in order to obtain a spatial path starting from this random position, it is necessary to obtain the position of the robotic arm through the machine vision system to determine the random position so that spatial path planning can be performed.

[0110] Reference Figure 8 As shown, as an embodiment of the present application, a spatial path planning is performed on a robotic arm according to a manipulation instruction to obtain a spatial path for controlling the robotic arm, including:

[0111] S232. Determine a number of path points in the space where the ship's control panel is located based on the outer dimensions, component positions, and relative adjustment amounts of the control components in the control instructions.

[0112] S234. Perform quaternion interpolation between several path points to obtain a spatial path passing through the path points.

[0113] Specifically, quaternion interpolation represents the rotation in space through a scalar and three imaginary parts, which can effectively represent the rotation of the manipulation component in space and achieve a smooth rotation transition, so that the robotic arm can smoothly transition the posture. In this embodiment, the contact point between the robotic arm and the manipulation component is determined according to the outer dimensions and component position of the manipulation component as the first path point. And the contact point between the robotic arm and the manipulation component after manipulation is determined according to the relative adjustment amount as the second path point, so that the spatial path for manipulating the manipulation component is obtained according to the first path point and the second path point. It can be understood that all manipulation components that need to be manipulated have corresponding spatial paths, and each spatial path is also connected by an inter-component spatial path, so as to achieve smooth movement of the robotic arm between different manipulation components.

[0114] Accordingly, please refer to Figure 9 An embodiment of the present application provides an auxiliary evaluation device for a transport device console, which is applied to a computer-assisted evaluation system. The computer-assisted evaluation system includes a machine vision system, a robotic arm, and a background computer. The console is used to control a simulated transport device during the evaluation process. The console is located within the viewing angle of the machine vision system and within the control range of the robotic arm. The machine vision system and the robotic arm are communicatively connected to the background computer, which is in turn communicatively connected to the console. The device includes:

[0115] A preset task determination module 910 is used to determine a preset navigation task for evaluating the preset required performance of the control console;

[0116] The control instruction planning module 920 is used to obtain the navigation status information of the simulated transport device and the control status information of the control panel through the machine vision system before executing the preset navigation task; and to plan the navigation instructions based on the navigation status information, the control status information and the preset navigation task, to obtain the control instructions for instructing the manipulator arm to operate the control panel;

[0117] The manipulation instruction execution module 930 is used to obtain mechanical load index data of the manipulator arm during the execution of the manipulation instruction, and to analyze the visual display state of the control panel through the machine vision system to obtain visual load index data. The mechanical load index data is used to describe the load on the operator's arm when the control panel is actually being operated, and the visual load index data is used to describe the load on the operator's eyes when the control panel is actually being operated.

[0118] The preset performance evaluation module 940 is used to perform a comprehensive evaluation based on the mechanical load index data and the visual load index data to obtain evaluation data of the control console in terms of the preset required performance.

[0119] In some optional embodiments, the transport device is a ship, the simulated transport device is a simulated ship, and the control console is a ship control console; the control instruction execution module 930 includes:

[0120] The control torque acquisition unit is used to obtain the control torque data of the robotic arm during the execution of the control instruction.

[0121] The repetition frequency statistics unit is used to count the number of times the manipulator arm manipulates the same manipulation component on the ship's control console during the execution of the manipulation instruction, and obtain the component repetition frequency.

[0122] The repeated action statistics unit is used to count the number of times the robot arm performs the same manipulation action during the execution of the manipulation instruction to obtain repeated manipulation data.

[0123] The mechanical load index data acquisition unit is used to obtain the mechanical load index data according to the control torque data, the component repetition frequency and the repeated control data.

[0124] In some optional implementations, the manipulation instruction execution module 930 further includes:

[0125] The visual range analysis unit is used to analyze the visual range of the ship control console according to the visual display status to obtain visual range data.

[0126] The visual feature analysis unit is used to perform visual feature analysis on the visual display state to obtain visual feature data; wherein the visual feature data includes data obtained by performing color analysis and brightness analysis on the visual display state.

[0127] The control state comparison unit is used to compare the control state information obtained by the machine vision system with the actual control state information of the ship control panel, and obtain error recognition data based on the difference between the control state information and the actual control state information.

[0128] The visual load index data acquisition unit is used to obtain visual load index data according to the visual field range data, the visual feature data and the error recognition data.

[0129] In some optional embodiments, the ship control console includes a display screen and a control component, and the visual display state includes a display screen display state and a control component display state; the control instruction execution module 930 further includes:

[0130] The navigation status information recognition unit is used to identify the information displayed on the display screen and obtain the navigation status information of the simulated ship.

[0131] The control state information identification unit is used to identify the control state of the control component based on the display state of the control component to obtain the control state information of the ship control panel.

[0132] In some optional implementations, the manipulation instruction planning module 920 includes:

[0133] The navigation instruction planning unit is used to plan navigation instructions for the simulated ship according to the navigation status information and the navigation target of the simulated ship in the preset navigation mission, and obtain navigation control instructions for controlling the simulated ship.

[0134] The maneuvering instruction planning unit is used to compare the navigation control instructions with the maneuvering status information of the ship's control panel, determine the relative adjustment amount of each control component on the ship's control panel, and obtain the maneuvering instructions.

[0135] In some optional implementations, the manipulation instruction planning module 920 further includes:

[0136] The spatial path planning unit is used to plan the spatial path of the robotic arm according to the manipulation instructions to obtain the spatial path used to control the robotic arm.

[0137] In some optional implementations, the manipulation instruction planning module 920 further includes:

[0138] The path point determination unit is used to determine a number of path points in the space where the ship's control panel is located according to the external dimensions, component positions and relative adjustment amounts of the control components in the control instructions.

[0139] The path point interpolation unit is used to perform quaternion interpolation between several path points to obtain a spatial path passing through the path points.

[0140] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0141] The auxiliary evaluation device of the transport device console in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0142] See also Figure 10 , Figure 101 is a structural diagram of a computer device provided by an embodiment of the present application. As shown in the figure, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 10 A processor 10 is taken as an example.

[0143] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0144] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0145] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0147] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0148] The embodiments of the present application also provide a computer-readable storage medium. The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0149] An embodiment of the present application provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method according to any embodiment of the present application.

[0150] Although the embodiments of the present application have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations shall fall within the scope defined by the appended claims.

[0151] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0152] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0153] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0157] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0158] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0159] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

[0160] Although the embodiments of the present application have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations shall fall within the scope defined by the appended claims.

Claims

1. A method for assisting evaluation of a transport device console, characterized in that: The method is applied to a computer-aided evaluation system, the computer-aided evaluation system including a machine vision system, a robotic arm, and a background computer; the console is used to control a simulated transport device during the evaluation process, the console is located within the shooting angle of the machine vision system and within the control range of the robotic arm, the machine vision system and the robotic arm are communicatively connected to the background computer, and the background computer is communicatively connected to the console; the method includes: determining a predetermined navigation task for evaluating a predetermined required performance of the control panel; Before executing the preset navigation task, obtaining navigation status information of the simulated transport device and manipulation status information of the control panel through the machine vision system; and performing navigation instruction planning based on the navigation status information, the manipulation status information, and the preset navigation task to obtain a manipulation instruction instructing the robotic arm to manipulate the control panel; Acquiring mechanical load index data of the robotic arm during the execution of the manipulation instruction, and analyzing the visual display state of the console through the machine vision system to obtain visual load index data; wherein the mechanical load index data is used to describe the load on the operator's arm when the console is actually manipulated, and the visual load index data is used to describe the load on the operator's eyes when the console is actually manipulated; A comprehensive evaluation is performed based on the mechanical load index data and the visual load index data to obtain evaluation data of the control panel in terms of the preset required performance.

2. The method according to claim 1, characterized in that The transport device is a ship, the simulated transport device is a simulated ship, and the control console is a ship control console; The mechanical load index data is obtained by: Acquiring manipulation torque data of the robotic arm during execution of the manipulation instruction; Counting the number of times the manipulator arm manipulates the same manipulation component on the ship control console during the execution of the manipulation instruction to obtain a component repetition frequency; Counting the number of times the robot arm performs the same manipulation action during the execution of the manipulation instruction to obtain repeated manipulation data; The mechanical load index data is obtained according to the control torque data, the component repetition frequency and the repeated control data.

3. The method according to claim 2, characterized in that The visual load index data is obtained by: Performing a visual range analysis on the ship control console according to the visual display state to obtain visual range data; Performing visual feature analysis on the visual display state to obtain visual feature data; wherein the visual feature data includes data obtained by performing color analysis and brightness analysis on the visual display state; comparing the manipulation state information acquired by the machine vision system with actual manipulation state information of the ship control console, and obtaining error recognition data according to a difference between the manipulation state information and the actual manipulation state information; The visual load index data is obtained according to the visual field range data, the visual feature data and the error recognition data.

4. The method according to claim 2, characterized in that The ship control console includes a display screen and a control component, and the visual display state includes a display screen display state and a control component display state; The visual display state is obtained by: Performing information recognition on the display screen display status to obtain navigation status information of the simulated ship; The control component display status is identified to obtain the control status information of the ship control panel.

5. The method according to claim 2, characterized in that The performing of navigation instruction planning according to the navigation state information, the manipulation state information, and the preset navigation task to obtain a manipulation instruction instructing the manipulator arm to manipulate the ship control console includes: performing navigation instruction planning for the simulated ship according to the navigation state information and the navigation target of the simulated ship in the preset navigation mission, and obtaining navigation control instructions for controlling the simulated ship; The navigation control instruction is compared with the control state information of the ship control panel, and the relative adjustment amount of each control component on the ship control panel is determined to obtain the control instruction.

6. The method according to claim 2, characterized in that Before obtaining the mechanical load index data of the robotic arm during the execution of the manipulation instruction, the method includes: A spatial path planning is performed on the robotic arm according to the manipulation instruction to obtain a spatial path for controlling the robotic arm.

7. The method according to claim 6, characterized in that The performing spatial path planning on the robotic arm according to the manipulation instruction to obtain a spatial path for controlling the robotic arm includes: Determining a plurality of path points in the space where the ship control panel is located according to the outer dimensions, component positions, and relative adjustment amounts of the control components in the control instructions; Quaternion interpolation is performed between the plurality of path points to obtain a spatial path passing through the path points.

8. An auxiliary evaluation device for a transport device console, characterized in that: The present invention is applied to a computer-aided evaluation system, the computer-aided evaluation system including a machine vision system, a robotic arm, and a background computer; the console is used to control the simulated transport device during the evaluation process, the console is located within the shooting angle of the machine vision system and within the control range of the robotic arm, the machine vision system and the robotic arm are communicatively connected to the background computer, and the background computer is communicatively connected to the console; the device includes: a preset task determination module, configured to determine a preset navigation task for evaluating the preset required performance of the control console; a manipulation instruction planning module, configured to obtain, through the machine vision system, navigation state information of the simulated transport device and manipulation state information of the control console before executing the preset navigation task; and perform navigation instruction planning based on the navigation state information, the manipulation state information, and the preset navigation task to obtain a manipulation instruction for instructing the robotic arm to manipulate the control console; a manipulation instruction execution module, configured to obtain mechanical load index data of the manipulator arm during the execution of the manipulation instruction, and to analyze the visual display state of the control panel through the machine vision system to obtain visual load index data; wherein the mechanical load index data is used to describe the load on the operator's arm when the control panel is actually being manipulated, and the visual load index data is used to describe the load on the operator's eyes when the control panel is actually being manipulated; The preset performance evaluation module is used to perform a comprehensive evaluation based on the mechanical load index data and the visual load index data to obtain evaluation data of the control console in terms of the preset required performance.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images