A multi-information fusion-based instrument operation correctness detection method and device and medium

Abstract

The application discloses a kind of instrument operation correctness detection method, device and medium based on multi-information fusion, it is related to instrument operation correctness detection technical field.The method includes obtaining the operation video information of the target instrument multi-view;Based on the operation video information of the target instrument, the operation point for operating the target instrument is tracked in real time, to obtain the real-time position information of the operation point;Obtain the sound information of the operation object on the target instrument;Based on the real-time position information of the operation point and the sound information of the operation object, it is judged whether the operation object is started;In the case where the operation object is started, the corresponding operation information of the operation object after being started is displayed;The corresponding operation information of the operation object after being started is compared with preset information, to obtain the detection result of the operation correctness of the target instrument.Through the above technical scheme, the correctness of the target instrument operation can be more accurately detected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of instrument operation correctness detection, and particularly relates to an instrument operation correctness detection method and device based on multi-information fusion and a medium. BACKGROUND

[0002] In production and test sites, there are a large number of instruments and equipment that need to be manually operated. The correctness of the operation of these instruments and equipment is not only related to the correctness of the instrument recorded data, but also related to the production quality, and even related to the safety of personnel. Therefore, the operation specifications of the instruments and equipment are formulated, and the pre-post training is conducted on the operators. However, for some discrete manufacturing flexible production lines, the operators often need to be familiar with the operation of a plurality of instruments and equipment, and some instruments and equipment are added every year, thereby increasing the difficulty of standard operation.

[0003] In summary, in the prior art, the operator only refers to the operation specification of the instrument, and cannot accurately detect the correctness of the instrument operation. SUMMARY

[0004] The main purpose of the present application is to provide an instrument operation correctness detection method and device based on multi-information fusion and a medium, and to solve the technical problem that the operator in the prior art only refers to the operation specification of the instrument, and cannot accurately detect the correctness of the instrument operation.

[0005] To achieve the above purpose, the first aspect of the present application provides an instrument operation correctness detection method based on multi-information fusion, which comprises:

[0006] Obtaining multi-view operation video information of a target instrument;

[0007] Based on the operation video information of the target instrument, the operation point for operating the target instrument is tracked in real time to obtain real-time position information of the operation point;

[0008] Obtaining sound information of an operation object on the target instrument; wherein the operation object is an object operated by the operation point;

[0009] Based on the real-time position information of the operation point and the sound information of the operation object, it is judged whether the operation object is started;

[0010] In the case that the operation object is started, the operation information corresponding to the operation object after being started is displayed;

[0011] The operation information corresponding to the operation object after being started is compared with preset information to obtain a detection result of the operation correctness of the target instrument.

[0012] Optionally, after the step of comparing the operation information corresponding to the operation object being started with preset information to obtain the detection result of the operation correctness of the target instrument, the method further comprises:

[0013] Based on the detection result of the operation correctness of the target instrument, an operation error of the target instrument is obtained.

[0014] Based on the operation error, an error level of the operation error is obtained.

[0015] Based on the error level of the operation error, prompt information is sent.

[0016] Optionally, after the step of sending prompt information based on the error level of the operation error, the method further comprises:

[0017] The error level of the operation error is compared with a preset level, wherein the preset level is used to determine the size of the operation error of the target instrument.

[0018] If the error level of the operation error is greater than the preset level, a cut-off instruction is sent, wherein the cut-off instruction is an instruction to cut off the output signal corresponding to the operation error.

[0019] Optionally, the step of comparing the operation information corresponding to the operation object being started with preset information to obtain the detection result of the operation correctness of the target instrument comprises:

[0020] The operation information corresponding to the operation object being started is output.

[0021] The output operation information corresponding to the operation object is recognized to extract preset information corresponding to the recognized operation information of the operation object.

[0022] The recognized operation information corresponding to the operation object is compared with the extracted preset information to obtain the detection result of the operation correctness of the target instrument.

[0023] Optionally, the step of determining whether the operation object is started based on the real-time position information of the operation point and the sound information of the operation object comprises:

[0024] Based on the real-time position information of the operation point, the position coordinates of the operation point are obtained.

[0025] Based on the operation video information of the target instrument, the position coordinates of the operation object of the target instrument are obtained.

[0026] Based on the sound information of the operation object, the position coordinates of the operation point, and the position coordinates of the operation object of the target instrument, it is determined whether the operation object is started.

[0027] Optionally, judging whether the operation object is activated based on the sound information of the operation object, the position coordinate of the operation point and the position coordinate of the operation object of the target instrument, comprises:

[0028] Obtaining a coordinate error based on the position coordinate of the operation point and the position coordinate of the operation object of the target instrument; the coordinate error is an absolute value of a difference between the position coordinate of the operation point and the position coordinate of the operation object of the target instrument;

[0029] Judging whether the operation object is activated based on a preset operation threshold, the sound information of the operation object and the coordinate error; wherein the coordinate error comprises the operation threshold.

[0030] Optionally, judging whether the operation object is activated based on the preset operation threshold, the sound information of the operation point and the coordinate error, comprises:

[0031] When the coordinate error is less than or equal to the operation threshold, and the sound information of the operation object is received, the operation object is activated.

[0032] Optionally, before the step of tracking the operation point operating the target instrument in real time based on the operation video information of the target instrument to obtain real-time position information of the operation point, further comprising:

[0033] Distributing the operation video information of the target instrument;

[0034] The step of tracking the operation point operating the target instrument in real time based on the operation video information of the target instrument to obtain real-time position information of the operation point, comprises:

[0035] Tracking the operation point operating the target instrument in real time based on the distributed operation video information of the target instrument to obtain real-time position information of the operation point.

[0036] Optionally, before the step of judging whether the operation object is activated based on the real-time position information of the operation point and the sound information of the operation object, further comprising:

[0037] Identifying the sound information of the operation object to obtain sound information of the operation object corresponding to being operated by the operation point;

[0038] The step of judging whether the operation object is activated based on the real-time position information of the operation point and the sound information of the operation object, comprises:

[0039] Judge whether the operation object is started based on the real-time position information of the operation point and the sound information of the operation object identified.

[0040] Optionally, the sound information of the operation object is identified to obtain the sound information of the operation object corresponding to the operation of the operation point.

[0041] Pre-learn the sound information of various operation objects on the target instrument to obtain pre-stored operation sound information of various operation objects.

[0042] Identify the sound information of the operation object based on the pre-stored operation sound information of various operation objects to obtain the sound information of the operation object corresponding to the operation of the operation point.

[0043] In a second aspect, the present application provides an instrument operation correctness detection device based on multi-information fusion, the device comprising:

[0044] An acquisition module is configured to acquire multi-view operation video information of a target instrument.

[0045] A first obtaining module is configured to track an operation point operating the target instrument in real time based on the operation video information of the target instrument to obtain real-time position information of the operation point.

[0046] A second obtaining module is configured to obtain sound information of an operation object on the target instrument, wherein the operation object is an object operated by the operation point.

[0047] A judging module is configured to judge whether the operation object is started based on the real-time position information of the operation point and the sound information of the operation object.

[0048] A display module is configured to display operation information corresponding to the operation of the operation object in the case that the operation object is started.

[0049] A third obtaining module is configured to compare the operation information corresponding to the operation of the operation object with preset information to obtain a detection result of operation correctness of the target instrument.

[0050] In a third aspect, the present application provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the method in the embodiments.

[0051] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the processor executes the computer program to realize the method in the embodiments.

[0052] By means of the technical solutions described above, the application has at least the following beneficial effects:

[0053] The method comprises the following steps: acquiring operation video information of a target instrument from multiple perspectives; tracking an operation point in real time based on the operation video information of the target instrument, to obtain real-time position information of the operation point; obtaining sound information of an operation object on the target instrument; determining whether the operation object is activated based on the real-time position information of the operation point and the sound information of the operation object; displaying operation information corresponding to the operation object after the operation object is activated, in a case where the operation object is activated; and comparing the operation information corresponding to the operation object after the operation object is activated with preset information, to obtain a detection result of operation correctness of the target instrument. That is, when it is necessary to detect the operation correctness of the target instrument, operation video information of the target instrument from multiple perspectives, such as key area video, display screen video and instrument accessory video, is obtained first, then real-time position information of the operation point is obtained based on the video information, and sound information of the operation object on the target instrument is obtained, and then whether the operation object is activated is determined according to the obtained real-time position information of the operation point and the sound information of the operation object. When the operation object is activated, operation information corresponding to the activated operation object is displayed on the target instrument, and then the operation information is compared with preset information obtained in advance, and finally a detection result of operation correctness of the target instrument is obtained according to the comparison. That is, more comprehensive information of the target instrument can be obtained from multiple perspectives, and the more comprehensive information guarantees more accurate information of the target instrument during operation, and the more accurate information is the basis for more accurate determination of operation correctness of the target instrument. Moreover, sound information of the operation object on the target instrument is obtained, and whether the operation object is activated can be more accurately determined in combination with the sound information of the operation object and the real-time position information of the operation point, and then the operation information after the operation object is activated is compared with preset information, and therefore the operation correctness of the target instrument can be more accurately detected. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 A computer device structure schematic diagram of a hardware running environment related to the embodiments of the application;

[0055] Figure 2 A flowchart of a method for detecting operation correctness of an instrument based on multi-information fusion according to an embodiment of the application;

[0056] Figure 3 A system framework schematic diagram for implementing the method provided by the present application is provided for the embodiments of the present application.

[0057] Figure 4 A schematic diagram of an instrument operation correctness detection device based on multi-information fusion for the embodiments of the present application.

[0058] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0059] It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0060] In production and test sites, there are a large number of instruments and equipment that need to be operated manually. The correctness of the operation of these instruments and equipment is not only related to the correctness of the instrument recording data, but also related to the production quality, and even the safety of personnel. Therefore, the operation specifications of the instruments and equipment are formulated, and the pre-service training of the operators is carried out. However, for some flexible production lines of discrete manufacturing industry, the operators often need to be familiar with the operation of multiple instruments and equipment, and some instruments and equipment are added every year, thereby increasing the difficulty of standard operation. On the other hand, the operation of the instruments and equipment is often related to the habits of the operators, inspection and specific circumstances, and the formulation of the operation specifications of the instruments and equipment inevitably requires all operators to perform only one operation step, which needs a lot of training to form muscle memory; and cannot cover all operation situations, such as re-inspection and fault elimination, which requires flexible and random operation. The above situations further increase the difficulty of standardizing and training the operation of the instruments.

[0061] The current instrument operation correctness detection method based on multi-information fusion cannot fully meet the requirements of some flexible production lines of discrete manufacturing industry. On the other hand, with the continuous development of artificial intelligence technology, it can be applied to the instrument operation correctness detection based on multi-information fusion. However, single video recognition cannot completely realize reliable detection of instrument operation, and the current video recognition technology can detect the finger movement of the operator and which instrument button or switch is covered, but due to the short operation stroke of the button action, the video cannot accurately determine whether the operator has pressed the button.

[0062] In summary, the current operators only refer to the operation specifications of the instruments, and cannot accurately detect the correctness of the instrument operation.

[0063] In order to solve the above technical problems, the present application provides an instrument operation correctness detection method, device and medium based on multi-information fusion. Before introducing the specific technical scheme of the present application, the hardware running environment involved in the embodiments of the present application is introduced.

[0064] Refer to Figure 1 , Figure 1 The computer device structure schematic diagram of the hardware running environment involved in the embodiment of the present application.

[0065] As Figure 1 shown, the computer device can include: a processor 1001, for example, a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between the components. The user interface 1003 can include a display screen, an input unit such as a keyboard, and an optional user interface 1003 can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 can be a high-speed random access memory (RAM) memory, or a stable non-volatile memory (NVM), such as a magnetic disk memory. The memory 1005 can also be an independent storage device from the aforementioned processor 1001.

[0066] Those skilled in the art can understand that Figure 1 the structure shown in the above description does not constitute a limitation on the computer device, and can include more or fewer components than the diagram, or combine certain components, or different component arrangements.

[0067] As Figure 1 shown, the memory 1005 as a storage medium can include an operating system, a data storage module, a network communication module, a user interface module, and an electronic program.

[0068] In Figure 1 the computer device shown in the above description, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the computer device of the present application can be arranged in the computer device, and the computer device calls the instrument operation correctness detection device based on multi-information fusion stored in the memory 1005 through the processor 1001, and executes the instrument operation correctness detection method based on multi-information fusion provided by the embodiment of the present application.

[0069] Refer to Figure 2Based on the hardware environment of the foregoing embodiments, the embodiments of the present application provide a multi-information fusion-based instrument operation correctness detection method, which comprises the following steps:

[0070] S10: Obtain multi-view operation video information of the target instrument.

[0071] In the specific implementation process, the target instrument refers to an instrument whose operation correctness needs to be detected, and the target instrument is a radio frequency signal source. The setting of the output signal includes parameters such as frequency, power, modulation mode, and debugging signal rate. For example, press the “FREQ” key to start setting the frequency, press the “ENTER” key, and the frequency setting is successful; press the “LEVEL” key to start setting the power, press the “ENTER” key, and the power setting is successful; press the “MOD” key to start setting the modulation mode and the modulation signal rate, press the “ENTER” key, and the modulation mode and the modulation signal rate setting are successful. Finally, press the “ON / OFF” key, and the instrument outputs the signal from the output port according to the set signal parameters, or stops outputting the signal. Multi-view refers to multiple aspects and multiple angles. Obtaining operation video information of the target instrument from multiple aspects and multiple angles is the basis for detecting the operation correctness of the target instrument. Specifically, the operation video information includes key area video information, display screen video information, and instrument accessory video information, etc. The operation action of the operation point (finger) is recognized through the key area video information, the numbers, symbols, and texts displayed by the instrument are recognized through the display screen video information, and the operation state of the accessory is recognized through the instrument accessory video information. A camera is directed at the target instrument to capture the operation process of the target instrument. The captured image covers all the keys and switches of the target instrument, so as to monitor the action of the operator's finger operating the keys and switches of the instrument. At the same time, the captured image covers the display on the front panel of the instrument, so as to monitor the display screen of the instrument. In addition, the captured image covers the necessary accessories of the instrument, so as to monitor the operation of the operator installing or removing the instrument accessories. The operation key audio of the target instrument is collected, and the keys and switch sounds of the target instrument are recognized through the sound characteristics. The output signal of the target instrument is collected, and the mode and parameters of the output signal are recognized. The operated button is confirmed through the operation action of the finger combined with the keys and switch sounds of the target instrument, the operation result of the target instrument is confirmed through the numbers, symbols, and texts displayed by the target instrument and the mode and parameters of the output signal, and the correctness of the instrument operation is judged through the operated button, the operation result of the target instrument, and the operation state of the accessory.

[0072] S11: Based on the operation video information of the target instrument, real-time tracking is performed on the operation point operating the target instrument to obtain real-time position information of the operation point.

[0073] In the specific implementation process, the operation point refers to a point at which the operation object can be started. Specifically, the operation point is preferably a finger, and the operation object is preferably a key and a switch of the target instrument. Specifically, the video motion recognition module adopts a three-dimensional hand key point real-time tracking detection algorithm. Through the algorithm of computer vision and deep learning, the coordinates (xfn, yfn) of the hand fingertip key point are intelligently positioned, the finger motion capture is realized, the finger operation action of the operator of the target instrument is recognized, and thus the real-time position information of the operation point (finger) is obtained.

[0074] S12: Obtain sound information of an operation object on the target instrument; wherein the operation object is an object operated by the operation point.

[0075] In the specific implementation process, the operation object (key or switch) can be started by the operation point (finger). The starting includes turning on and turning off. Specifically, as shown in Figure 3 The pickup is preferably attached to the operation panel of the target instrument, and the microphone of the pickup faces the operation panel of the target instrument, so as to improve the sensitivity of the sound detection of the key and the switch and reduce the influence of the environmental noise. The audio signal when the key is pressed and the switch is turned can be more accurately obtained through the pickup, and the more accurate audio information of the operation object is more conducive to the more accurate detection of the target instrument.

[0076] S13: Determine whether the operation object is started based on the real-time position information of the operation point and the sound information of the operation object.

[0077] In the specific implementation process, based on the real-time position information of the operation point, it can be known whether the operation point is located at the operation object. Based on the sound information of the operation object, it can be determined whether the operation point located at the operation object starts the operation object, that is, the operation point is not only located at the operation object, but also the operation object generates sound information, which indicates that the operation object is started. Through the double combination of the visual information (real-time position information of the operation point) and the sound information (sound information of the operation object), it can be more accurately determined whether the operation object is started.

[0078] S14: Display the operation information corresponding to the operation object after the operation object is started.

[0079] In the implementation process, when the operation object is determined to be activated, the operation information corresponding to the operation object is displayed. Specifically, for example, the "Confirm" key is pressed, and after the "Confirm", "ENTER", or "ENT" key indicating the confirmation operation is recognized, the display screen symbol detection is performed. At this time, the target instrument displays the result screen after the "Confirm" operation, and the display screen symbol detection module uses the display screen symbol automatic detection algorithm to recognize the numbers, symbols, and texts displayed by the target instrument, and outputs the display screen after the "Confirm" key is pressed and the recognition data of the numbers, symbols, and texts of the screen.

[0080] S15: The operation information corresponding to the operation object after the operation object is activated is compared with the preset information to obtain the detection result of the operation correctness of the target instrument.

[0081] In the implementation process, the preset information refers to the information set in advance, that is, the correct information corresponding to each key after the key is pressed is stored in advance, and the preset information is obtained. Specifically, the instrument screen and the recognition data of the numbers, symbols, and texts of the screen sent by the screen symbol detection module are compared with the required result screen of the operation steps and the corresponding numbers, symbols, and texts stored in advance, to realize the confirmation of the operation of the target instrument. Figure 3 In the example, the test item requires that the frequency is set to 120MHz, and then the "FREQ", "1", "2", "0", "M / μV", and "ENTER" keys of the target instrument are pressed in sequence. After the "ENTER" key is pressed, the display screen has the "FREQ 120.000000MHz" text and numbers. If the comparison is successful, the operation of the target instrument is confirmed. In the key operation recognition, the special key processing capability such as "BACKSPACE" is provided. For example, in the above example, the "FREQ", "1", "2", "1", "BACKSPACE", "0", "M / μV", and "ENTER" keys are pressed, and the error operation "1" is deleted. In this way, since the preset information is the correct information corresponding to the operation object, by comparing with the preset information, the detection result of the operation correctness of the target instrument is more accurate.

[0082] In summary, when it is necessary to detect the operation correctness of the target instrument, first, the operation video information of the target instrument under multiple perspectives is obtained, such as the key area video, the display screen video and the instrument accessory video, and then the real-time position information of the operation point is obtained based on the video information, and the sound information of the operation object on the target instrument is obtained, and then whether the operation object is started is judged according to the real-time position information of the operation point and the sound information of the operation object. When the operation object is started, the operation information corresponding to the started operation object is displayed on the target instrument, and then the operation information is compared with the preset information obtained in advance, and finally the detection result of the operation correctness of the target instrument is obtained according to the comparison. That is, since the method can obtain more comprehensive information of the target instrument from multiple perspectives, the more comprehensive information is the guarantee of obtaining more accurate information of the target instrument operation, and the more accurate information is the basis for finally more accurately judging the operation correctness of the target instrument. Moreover, the sound information of the operation object on the target instrument is also obtained, and the sound information of the operation object and the real-time position information of the operation point can be combined to more accurately judge whether the operation object is started, and then the operation information after the operation object is started is compared with the preset information, and the preset information is the correct information displayed by the target instrument after each operation object is operated, so that the operation correctness of the target instrument can be more accurately detected.

[0083] In order to more timely handle the operation error of the target instrument, in some embodiments, after the step of comparing the operation information corresponding to the started operation object with the preset information to obtain the detection result of the operation correctness of the target instrument, the method further comprises the steps of: first, obtaining the operation error of the target instrument based on the detection result of the operation correctness of the target instrument; then, obtaining the error level of the operation error based on the operation error; and finally, sending a prompt information based on the error level of the operation error.

[0084] In this embodiment, after the display information corresponding to the started operation object is compared with the preset information, the difference between the display information and the preset information is the operation error, and the operation error has large and small, and the operation error can be classified according to the size of the operation error, so as to obtain the error level. Specifically, after judging the operation correctness, the fusion processing machine also disposes the misoperation. The fusion processing machine is built-in with a misoperation level rule library, and the level of the misoperation is judged according to the rule library setting. The size of the error can be divided into different ranges, and each range corresponds to a corresponding error level, so that more accurate error level can be more easily obtained.

[0085] In order to better solve the harm caused by the operation error, in some embodiments, the step of sending prompt information based on the error level of the operation error further comprises: comparing the error level of the operation error with a preset level; wherein the preset level is used to determine the size of the operation error of the target instrument; and if the error level of the operation error is greater than the preset level, a cut-off instruction is sent; wherein the cut-off instruction is an instruction to cut off the output signal corresponding to the operation error.

[0086] In this embodiment, the preset level is a pre-set operation error level, which is equivalent to an operation error threshold. That is, when the error level of the operation error is greater than the preset level, it means that the operation error after pressing the operation object is large, and if it is not stopped in time, it may cause a large accident. Therefore, if the error level of the operation error is greater than the preset level, a cut-off instruction is sent to stop the related program and cut off the related operation. That is, general misoperation is prompted, misoperation that leads to safe production is alarmed, and the error output signal is cut off through a relay. In this way, the operation error is processed in layers, which can more flexibly handle the corresponding operation error and reduce the corresponding loss.

[0087] In order to more accurately obtain the detection result of the operation correctness of the target instrument, in some embodiments, the step of comparing the operation information corresponding to the started operation object with the preset information to obtain the detection result of the operation correctness of the target instrument comprises: first outputting the operation information corresponding to the started operation object; then identifying the output operation information corresponding to the operation object to extract the preset information corresponding to the identified operation information of the operation object; and finally comparing the identified operation information corresponding to the operation object with the extracted preset information to obtain the detection result of the operation correctness of the target instrument.

[0088] In the embodiment, when the operation object is started, the operation information corresponding to the operation object is displayed on the target instrument or other device, the operation information is output, and then the output operation information is identified, that is, it is identified that the operation information corresponds to which operation object. After the operation information is identified, the correct operation information corresponding to the identified operation object is compared with the output operation information, and the detection structure is obtained after the comparison. Specifically, the signal collector is used to collect the signal output by the target instrument, and the selection of the signal collector is related to the type of the signal output by the target instrument. In the example, the target instrument is a radio frequency signal source, the output signal is a radio frequency signal, and the corresponding signal collector is a radio frequency signal collector, which collects signal waveforms, frequencies, powers and other parameters. The signal processor uses a signal pattern recognition algorithm to recognize the pattern of the signal output by the target instrument, and also recognizes parameters including frequency, power and modulation spectrum characteristics (such as single carrier, AM, FM, etc.), so as to determine whether the correct operation is performed. When it is identified that the user presses the key representing the output meaning such as "output" and "ON / OFF", the judgment of the correctness of the output signal is started. Figure 3 As shown in the figure, in the embodiment, the test project requires that the output frequency is 120MHz and the power is -30dBm, and the single carrier is required. Therefore, the signal processor automatically measures whether the frequency, power and modulation spectrum characteristics of the signal meet the requirements of the test project. By identifying the operation object corresponding to the operation information, and then obtaining the corresponding preset information through the operation object, it can be more accurately known which operation object is started to generate the operation information, and finally it is more beneficial to detect the correctness of the operation of the target instrument.

[0089] In order to more accurately determine whether the operation object is started, in some embodiments, the step of determining whether the operation object is started based on the real-time position information of the operation point and the sound information of the operation object includes: first, obtaining the position coordinates of the operation point based on the real-time position information of the operation point; then, obtaining the position coordinates of the operation object of the target instrument based on the operation video information of the target instrument; and finally, determining whether the operation object is started based on the sound information of the operation object, the position coordinates of the operation point and the position coordinates of the operation object of the target instrument.

[0090] In the embodiment, the real-time position information of the operation point is captured by the plurality of cameras in real time, and a coordinate system is established in advance in the key area, so that each key has a corresponding coordinate, and the center coordinate of each key is (xn, yn). In the embodiment, the coordinate of the "FREQ" key is (xl, yl), the coordinate of the "LEVEL" key is (xl, y2), and so on. Based on the coordinate, the operation point also has a corresponding coordinate. In the example, the coordinates of the right hand fingertip key points are (xf1, yf1), (xf2, yf2), (xf3, yf3), (xf4, yf4), and (xf5, yf5) from the thumb, index finger, middle finger, ring finger, and little finger, respectively. The left hand is numbered as 6-10. When the operation point operates the corresponding key, the coordinate of the key is the coordinate of the operation point. When the coordinates of the operation point and the operation object are obtained, it can be accurately known whether the operation point is located at the operation object, and in combination with the sound of the operation object, it can be known whether the operation point starts the corresponding operation object. Specifically, the coordinate of the operation point coincides with the coordinate of the operation object, and the sound information of the operation object is obtained, which indicates that the operation point is started.

[0091] In order to more accurately obtain whether the operation object is started, in some embodiments, based on the sound information of the operation object, the position coordinate of the operation point, and the position coordinate of the operation object of the target instrument, the step of judging whether the operation object is started includes: first, based on the position coordinate of the operation point and the position coordinate of the operation object of the target instrument, obtaining a coordinate error; the coordinate error is the absolute value of the difference between the position coordinate of the operation point and the position coordinate of the operation object of the target instrument; then, based on a preset operation threshold, the sound information of the operation object, and the coordinate error, judging whether the operation object is started; wherein, the coordinate error includes the operation threshold.

[0092] In the embodiment, the operation threshold refers to a set coordinate error. Since the coordinates of the operation object are established based on the center of the operation object, even if the coordinates of the operation point do not completely coincide with the coordinates of the operation object, the operation point can still start the operation object. Therefore, the operation object can still be started within a certain coordinate error. The coordinate error can be regarded as the absolute value of the difference between the position coordinates of the operation point and the position coordinates of the operation object of the target instrument. Then, a special coordinate error, i.e., an operation error, is set in advance. As long as the coordinate error is within the operation threshold, it is considered that the operation point can start the operation object. Specifically, two modalities of video and audio are used, i.e., the output three-dimensional hand key points are used to track and detect the position in real time, and the output keys and switches are used to confirm the operation, so as to assist in judging whether the keys are pressed or the switches are toggled. In the example, when |(xfn, yfn)-(xn, yn)|≤key threshold and there is a key sound, it is judged that the key is pressed. The key threshold can be set according to actual needs, wherein xfn represents the x coordinate of the operation object, yfn represents the y coordinate of the operation object, xn represents the x coordinate of the operation point, and yn represents the y coordinate of the operation point. That is, when the coordinate error is less than or equal to the operation threshold, and the sound information of the operation object is received, the operation object is started.

[0093] In order to more accurately obtain the operation video information corresponding to the real-time position information of the operation point, in some embodiments, before the step of tracking the operation point operating the target instrument in real time based on the operation video information of the target instrument to obtain the real-time position information of the operation point, the operation video information of the target instrument is further allocated.

[0094] The step of tracking the operation point operating the target instrument in real time based on the operation video information of the target instrument to obtain the real-time position information of the operation point includes: tracking the operation point operating the target instrument in real time based on the allocated operation video information of the target instrument to obtain the real-time position information of the operation point.

[0095] In the embodiment, the operation video information of the target instrument obtained under multiple perspectives includes multiple types of operation video information. After the multiple types of operation video information are allocated, the operation video information related to the operation point can be allocated, and the real-time position information of the operation point can be obtained based on the allocated operation video information related to the operation point. In this way, the efficiency of obtaining the real-time position information of the operation point can be greatly improved. Specifically, in combination with the above-mentioned embodiment, the operation video information of the target instrument obtained under multiple perspectives includes multiple types of operation video information, and the multiple types of operation video information are allocated. Figure 3As shown, the camera video is sent to a video processor, and in the video processor, it is divided into three paths, and sent to different processing function modules respectively. The video of the button area is sent to an action recognition module, the video of the display screen is sent to a screen symbol detection module, and the video of the instrument accessory area is sent to a video image recognition module.

[0096] In order to obtain more accurate sound information of the operation object, in some embodiments, before the step of judging whether the operation object is activated based on the real-time position information of the operation point and the sound information of the operation object, the method further comprises: identifying the sound information of the operation object to obtain the sound information of the operation object corresponding to the operation of the operation point.

[0097] The step of judging whether the operation object is activated based on the real-time position information of the operation point and the sound information of the operation object comprises: judging whether the operation object is activated based on the real-time position information of the operation point and the identified sound information of the operation object.

[0098] In this embodiment, when the operation object is triggered by the operation point, the operation object will emit sound, but some noisy sound will inevitably be emitted in the surrounding environment. Therefore, the sound information emitted by the operation object needs to be identified. The sound information of the operation object identified can be used to more accurately judge whether the operation object is activated.

[0099] In order to more accurately identify the sound information of the operation object, in some embodiments, the step of identifying the sound information of the operation object to obtain the sound information of the operation object corresponding to the operation of the operation point comprises: first pre-learning the sound information of various operation objects on the target instrument to obtain pre-stored operation sound information of various operation objects; and then identifying the sound information of the operation object based on the pre-stored operation sound information of various operation objects to obtain the sound information of the operation object corresponding to the operation of the operation point.

[0100] In this embodiment, in order to more quickly and accurately identify the sound information of the operation object, the sound information of the operation object is first identified, and when the sound information emitted by the operation object is detected, it can be more quickly and accurately identified. Specifically, the key and switch audio collected by the sound collector is sent to the audio processor, and the audio processor uses sound feature recognition to identify the key and switch sound of the target instrument. Because different target instruments have different key and switch sounds, on the other hand, in a noisy environment, the key sound and switch sound are relatively light and easy to be disturbed by environmental noise. In order to improve the recognition rate of the instrument key and switch sound and eliminate the interference of environmental noise and noise, the audio processor needs to learn the key and switch sound in advance. As for how to learn, those skilled in the art are aware of it.

[0101] In order to realize the above method, a multi-modal fusion based multi-information fusion instrument operation correctness detection system is adopted, which comprises a multi-modal acquisition subsystem, a multi-modal recognition subsystem, a fusion processor and a relay. The multi-modal acquisition subsystem comprises a camera, a sound collector and a signal collector. The camera collects the operation video of the instrument, the sound collector collects the operation key audio of the instrument, and the signal collector collects the output signal of the instrument.

[0102] The multi-modal recognition subsystem comprises a video processor, an audio processor and a signal processor. The relay performs on-off control on the output signal of the instrument.

[0103] Specifically, the video processor comprises a processor, a memory, a device interface and a network interface. The device interface of the video processor is connected with the camera. The processor of the video processor performs video action recognition, picture symbol detection and video image recognition. The processor of the video processor comprises a video action recognition module, a display picture symbol detection module and a video image recognition module. The video processor divides the operation video into key area video, display picture video and instrument accessory video, and sends the divided video to the video action recognition module, the display picture symbol detection module and the video image recognition module for processing. The memory of the video processor stores instrument key coordinates, action library and processing results, instrument display symbol library and processing results, and instrument accessory library and processing results. The network interface of the video processor is connected with the fusion processor.

[0104] Specifically, the audio processor comprises a processor, a memory, a device interface and a network interface. The device interface of the audio processor is connected with the sound collector. The processor of the audio processor performs key and switch operation sound feature recognition. The memory of the audio processor stores key and switch feature audio library and processing results. The network interface of the audio processor is connected with the fusion processor.

[0105] Specifically, the signal processor includes a processor, a memory, a device interface and a network interface; the device interface of the signal processor is connected to the signal collector, the processor of the signal processor performs identification of the instrument output signal, the memory of the signal processor stores the processing results, and the network interface of the signal processor is connected to the fusion processor.

[0106] Specifically, the fusion processor includes a processor, a memory, a network interface and a device interface; the processor of the fusion processor performs multimodal fusion processing, and performs fusion processing on the recognition data of video action recognition, screen symbol detection, video image recognition, sound feature recognition and instrument signal recognition sent by the video processor, audio processor and signal processor; the memory of the fusion processor stores the comparison data; the network interface of the fusion processor is connected to the video processor, audio processor and signal processor to obtain video, audio and signal recognition data; the network interface of the fusion processor is also connected to the test process system to obtain test items and steps; the device interface of the fusion processor is connected to the relay to realize the control of the instrument transmission signal.

[0107] Specifically, the camera, microphone and signal collector adopt universal equipment.

[0108] To sum up, the instrument operation correctness detection system based on multi-information fusion of multimodal fusion in the embodiment of the present application simultaneously collects multimodal information such as video, audio and instrument output signals, and adopts multiple modes of recognition including action recognition, video image recognition, display screen symbol detection, sound feature recognition and instrument signal recognition. When using a multimodal fusion method, it performs instrument operation correctness detection based on multi-information fusion, utilizes the complementarity of multiple information sources, and improves the reliability of instrument operation correctness recognition; at the same time, it judges the level of erroneous operation and issues an alarm according to the level, which is more practical.

[0109] In another embodiment, Figure 4 As shown, based on the same inventive concept as the above embodiment, the embodiment of the present application further provides an instrument operation correctness detection device based on multi-information fusion, the device comprising:

[0110] An acquisition module is used to obtain multi-view operation video information of the target instrument;

[0111] A first obtaining module is configured to track an operating point of the target instrument in real time based on the operation video information of the target instrument to obtain real-time position information of the operating point;

[0112] A second obtaining module is configured to obtain sound information of an operating object on the target device; wherein the operating object is an object operated by the operating point;

[0113] a determining module, configured to determine whether the operation object is activated based on the real-time position information of the operation point and the sound information of the operation object;

[0114] a displaying module, configured to display operation information corresponding to the operation object after the operation object is activated if the operation object is activated;

[0115] a third obtaining module, configured to compare the operation information corresponding to the operation object after the operation object is activated with preset information to obtain a detection result of operation correctness of the target instrument.

[0116] It should be noted that the modules in the instrument operation correctness detection device based on multi-information fusion in the embodiment correspond to the steps in the instrument operation correctness detection method based on multi-information fusion in the foregoing embodiment one by one, and therefore, the specific implementation and the technical effects achieved by the embodiment can refer to the implementation of the foregoing instrument operation correctness detection method based on multi-information fusion, which will not be described herein again.

[0117] In addition, in an embodiment, the present application further provides a computer device, which comprises a processor, a memory and a computer program stored in the memory, and the computer program realizes the method in the foregoing embodiment when executed by the processor.

[0118] In addition, in an embodiment, the present application further provides a computer storage medium, which stores a computer program, and the computer program realizes the method in the foregoing embodiment when executed by a processor.

[0119] In some embodiments, the computer readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc or CD-ROM, etc., or various devices comprising one or any combination of the above memories. The computer can be various computing devices including a smart terminal and a server.

[0120] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0121] As an example, executable instructions can correspond to a file in a file system, can be stored in a part of a file that is used by the operating system to store application program data, can be stored as an "applet" in a general purpose software application, can be stored as a "plugin" in a web browser, or can be stored as an "app" in a mobile device, to name but a few.

[0122] As an example, executable instructions can be deployed to be executed on one computer, or on multiple computers of a system of computers in one location, or on multiple computers of a system of computers distributed among multiple locations and interconnected by a communication network.

[0123] It has to be noted that, as used herein, the terms "includes" and / or "contains", or any other tautological variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not limited to those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without further constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0124] The above-mentioned sequence numbers of the embodiments of the present application are only for description, and do not represent advantages or disadvantages of the embodiments.

[0125] Those skilled in the art can clearly understand the above-mentioned embodiment method from the description of the above embodiments, which can be realized by software and necessary general hardware platform, of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, an optical disk), and includes a plurality of instructions for causing a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0126] The above are only preferred embodiments of the present application, and do not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the contents of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for detecting the correctness of instrument operation based on multi-information fusion, characterized in that: The method comprises: Obtain multi-view operation video information of the target instrument; Based on the operation video information of the target instrument, an operation point of the target instrument is tracked in real time to obtain real-time position information of the operation point; Obtaining sound information of an operation object on the target device; wherein the operation object is an object operated by the operation point; determining whether the operation object is activated based on the real-time position information of the operation point and the sound information of the operation object; When the operation object is activated, displaying the operation information corresponding to the activation of the operation object; The operation information corresponding to the operation object after being activated is compared with the preset information to obtain the detection result of the operation correctness of the target instrument; including: outputting the operation information corresponding to the activated operation object; identifying the operation information corresponding to the output operation object to extract the preset information corresponding to the operation information of the identified operation object; comparing the operation information corresponding to the identified operation object with the extracted preset information to obtain the detection result of the operation correctness of the target instrument.

2. The instrument operation correctness detection method based on multi-information fusion according to claim 1, characterized in that: After the step of comparing the operation information corresponding to the operation object after being activated with the preset information to obtain a test result of the correctness of the operation of the target instrument, the method further includes: obtaining an operation error of the target instrument based on a detection result of the operation correctness of the target instrument; Based on the operation error, obtaining an error level of the operation error; Based on the error level of the operation error, a prompt message is sent.

3. The instrument operation correctness detection method based on multi-information fusion according to claim 2, characterized in that: After the step of sending a prompt message based on the error level of the operation error, the method further includes: Comparing the error level of the operation error with a preset level; wherein the preset level is used to determine the magnitude of the operation error of the target instrument; If the error level of the operation error is greater than the preset level, a cut-off instruction is sent; wherein the cut-off instruction is an instruction to cut off the output signal corresponding to the operation error.

4. The instrument operation correctness detection method based on multi-information fusion according to claim 1, characterized in that: The determining whether the operation object is activated based on the real-time position information of the operation point and the sound information of the operation object includes: Based on the real-time position information of the operating point, obtaining the position coordinates of the operating point; obtaining the position coordinates of an operation object of the target instrument based on the operation video information of the target instrument; Based on the sound information of the operation object, the position coordinates of the operation point, and the position coordinates of the operation object of the target instrument, it is determined whether the operation object is activated.

5. The instrument operation correctness detection method based on multi-information fusion according to claim 4 is characterized in that: The determining whether the operation object is activated based on the sound information of the operation object, the position coordinates of the operation point, and the position coordinates of the operation object of the target device includes: Obtaining a coordinate error based on the position coordinates of the operating point and the position coordinates of the operating object of the target instrument; the coordinate error being an absolute value of a difference between the position coordinates of the operating point and the position coordinates of the operating object of the target instrument; Based on a preset operation threshold, the sound information of the operation object and the coordinate error, it is determined whether the operation object is activated; wherein the coordinate error includes the operation threshold.

6. The instrument operation correctness detection method based on multi-information fusion according to claim 5, characterized in that: The determining whether the operation object is activated based on a preset operation threshold, the sound information of the operation point, and the coordinate error includes: When the coordinate error is less than or equal to the operation threshold and the sound information of the operation object is received, the operation object is activated.

7. The instrument operation correctness detection method based on multi-information fusion according to claim 1, characterized in that: Before the step of tracking the operating point of the target instrument in real time based on the operation video information of the target instrument to obtain the real-time position information of the operating point, the method further includes: distributing operation video information of the target instrument; The real-time tracking of the operating point of the target instrument based on the operation video information of the target instrument to obtain the real-time position information of the operating point includes: Based on the allocated operation video information of the target instrument, the operation point of the target instrument is tracked in real time to obtain real-time position information of the operation point.

8. The method for detecting correctness of instrument operation based on multi-information fusion according to claim 1, wherein: Before the step of determining whether the operation object is activated based on the real-time position information of the operation point and the sound information of the operation object, the method further includes: Recognizing the sound information of the operation object to obtain the sound information of the operation object corresponding to the operation point; The determining whether the operation object is activated based on the real-time position information of the operation point and the sound information of the operation object includes: Based on the real-time position information of the operation point and the recognized sound information of the operation object, it is determined whether the operation object is activated.

9. The method for detecting correctness of instrument operation based on multi-information fusion according to claim 8, characterized in that: The identifying the sound information of the operation object to obtain the sound information of the operation object corresponding to the operation point operation includes: Pre-learning the sound information of various operation objects on the target instrument to obtain pre-stored operation sound information of various operation objects; Based on the pre-stored operation sound information of various operation objects, the sound information of the operation object is identified to obtain the sound information of the operation object corresponding to the operation of the operation point.

10. An instrument operation correctness detection device based on multi-information fusion, characterized in that: The device comprises: An acquisition module is used to obtain multi-view operation video information of the target instrument; A first obtaining module is configured to track an operating point of the target instrument in real time based on the operation video information of the target instrument to obtain real-time position information of the operating point; A second obtaining module is configured to obtain sound information of an operating object on the target device; wherein the operating object is an object operated by the operating point; a judgment module, configured to judge whether the operation object is activated based on the real-time position information of the operation point and the sound information of the operation object; A display module, configured to display operation information corresponding to the operation object being activated when the operation object is activated; The third acquisition module is used to compare the operation information corresponding to the operation object after it is activated with the preset information to obtain the detection result of the operation correctness of the target instrument; it includes: outputting the operation information corresponding to the activated operation object; identifying the operation information corresponding to the output operation object to extract the preset information corresponding to the operation information of the identified operation object; comparing the operation information corresponding to the identified operation object with the extracted preset information to obtain the detection result of the operation correctness of the target instrument.

11. A computer device, characterized in that: The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the processor executes the computer program to implement the method according to any one of claims 1 to 9.

Images