Visual device access management method and device, electronic device and storage medium

Abstract

The application provides a visual equipment access management method and device, electronic equipment and a storage medium. The method is applied to the electronic equipment and includes the following steps: obtaining an interface protocol and equipment information of the visual equipment; matching a protocol specification according to the interface protocol; the protocol specification is used for translating data electrical signals sent by the visual equipment into readable data; establishing a data transmission thread and a data processing thread according to the equipment information; the data transmission thread is used for distributing the readable data, and the data processing thread is used for sending the readable data distributed by the data transmission thread to an application program or a memory after processing; establishing an equipment abstract model according to the protocol specification, the data transmission thread and the data processing thread; and the equipment abstract model is used for realizing data interaction between the visual equipment and the application program or the memory. According to different visual equipment, the application establishes a corresponding equipment abstract model, and the compatibility between the hardware equipment and the software program can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of communication, and particularly relates to a visual device access management method and device, an electronic device and a storage medium. BACKGROUND

[0002] At present, machine vision technology has been widely applied in multiple industries. Visual devices usually need to be connected to electronic devices to enable the electronic devices to obtain target data (such as image data, point cloud data, etc.) through the visual devices. Since different visual devices use different data transmission interfaces, the electronic devices need to download or install corresponding protocols and driver software to assist the normal data communication between the electronic devices and the visual devices when obtaining target data from different visual devices. SUMMARY

[0003] The present application aims to provide a visual device access management method and device, an electronic device and a storage medium, which aims to improve the compatibility of communication between electronic devices and different visual devices.

[0004] The first aspect of the present application provides a visual device access management method, which is applied to an electronic device and includes the following steps: obtaining an interface protocol and device information of a visual device; matching a protocol specification according to the interface protocol; the protocol specification is used to translate data electrical signals sent by the visual device into readable data; establishing a data transmission thread and a data processing thread according to the device information; the data transmission thread is used to distribute the readable data, and the data processing thread is used to process the readable data distributed by the data transmission thread and then send the processed data to an application program or a memory; establishing a device abstraction model according to the protocol specification, the data transmission thread and the data processing thread; the device abstraction model is used to realize data interaction between the visual device and the application program or the memory.

[0005] Further, the visual device access management method further includes the following steps: establishing a signal compiling thread according to the running environment of the electronic device; the compiling thread is used to translate control instructions sent by the application program into control electrical signals; and the compiling thread is added to the device abstraction model.

[0006] The second aspect of the embodiment of the present application provides a visual equipment access management device, which is applied to an electronic device and comprises: an information acquisition module, configured to acquire an interface protocol and device information of a visual equipment; a specification matching module, configured to match a protocol specification according to the interface protocol; the protocol specification is used to translate data electrical signals sent by the visual equipment into readable data; a thread establishing module, configured to establish a data transmission thread and a data processing thread according to the device information; the data transmission thread is used to distribute the readable data, and the data processing thread is used to send the readable data distributed by the data transmission thread to an application program or a memory after processing; a model establishing module, configured to establish a device abstract model according to the protocol specification, the data transmission thread and the data processing thread; and the device abstract model is used to realize data interaction between the visual equipment and the application program or the memory.

[0007] The third aspect of the embodiment of the present application provides an electronic device, comprising a processor, a device interface connected with the processor and a memory; the device interface is configured to establish a communication connection with a visual equipment; the memory is configured to store a computer program capable of running on the processor; and the processor is configured to implement the steps of the method according to the first aspect when executing the computer program.

[0008] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program; when the computer program is executed by a processor, the steps of the method according to the first aspect are implemented.

[0009] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0010] In the embodiment of the present application, the visual equipment access management method is applied to an electronic device, a device abstract model is constructed by acquiring an interface protocol and device information of a visual equipment, matching a protocol specification, establishing a data transmission thread and a data processing thread, and data interaction between the visual equipment and the electronic device can be translated and delivered by the device abstract model. According to different visual equipment to be accessed, the corresponding device abstract model is established, which can improve the compatibility between hardware devices and software programs. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 The connection relationship between the visual equipment and the electronic device provided by an embodiment of the present application is shown in the schematic diagram.

[0013] Figure 2 A flowchart of a visual equipment intervention management method provided by an embodiment of the present application is shown in FIG. 1.

[0014] Figure 3 A connection relationship diagram of a visual equipment and an electronic device provided by another embodiment of the present application is shown in FIG. 2.

[0015] Figure 4 A hierarchical diagram of a computer program corresponding to the visual equipment access management method provided by an embodiment of the present application is shown in FIG. 3.

[0016] Figure 5 A flowchart of a visual equipment access management method provided by another embodiment of the present application is shown in FIG. 4.

[0017] Figure 6 A structure diagram of a visual equipment access management apparatus provided by an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0018] In order to make the technical problems to be solved by the present application, technical solutions and beneficial effects more clearly understood, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0019] Figure 1 A connection relationship diagram of a visual equipment and an electronic device provided by an embodiment of the present application is shown in FIG. 1. In the present embodiment, the electronic device 110 includes a device interface 111, a processor 112 and a memory 113, and the visual equipment 120 establishes a connection with the electronic device 110 through the device interface 111. The memory 113 is configured to store a computer program capable of running on the processor 112; and the processor 112 is configured to implement at least the steps of the visual equipment access management method when executing the computer program, or implement at least the functions of the modules in the visual equipment access management apparatus embodiment when executing the computer program.

[0020] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 113 and executed by the processor 112 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device 110.

[0021] The electronic device 110 can be a computing device such as a desktop computer, a notebook computer, a palm computer, and a cloud server. The electronic device 110 can include, but is not limited to, a processor, a memory, and the like. The vision device 120 can be at least one of an RGB camera, a depth camera (such as a structured light camera, a binocular camera, an iToF camera, a dToF camera), and the like. The vision device 120 can include, but is not limited to, an optical element, an optical sensor, and the like. For example, the structured light camera can include an infrared (IR) sensor, a depth processor, an RGB sensor, and a pose sensor; the structured light camera can include an IR sensor, a depth processor, and an RGB sensor; the binocular camera can include a left IR sensor, a right IR sensor, a depth processor, and an RGB sensor. Of course, in actual applications, the vision device 120 can also include more or fewer devices. Those skilled in the art can understand that Figure 1 The electronic device and the vision device in the above are merely examples of the electronic device 110 and the vision device 120, and do not constitute a limitation on the electronic device 110 and the vision device 120, and can include more or fewer components than the illustration, or combine certain components, or different components, for example, the electronic device can also include an input / output device, a network access device, a bus, and the like.

[0022] The device interface 111 can be a universal serial bus (USB) interface, a Bluetooth transmission interface, an Ethernet interface, an HDMI interface, a UART interface, a ModBus interface, and the like. The device interface 111 is used to obtain the electrical signal sent by the vision device to the electronic device, and the device interface 111 is also used to obtain the control signal sent by the electronic device to the vision device. The specific structure of the device interface 111 is not limited here.

[0023] The processor 112 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0024] The memory 113 can be an internal storage unit of the electronic device 110, for example, a hard disk or a memory of the electronic device 110. The memory 113 can also be an external storage device of the electronic device 110, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 110. Further, the memory 113 can include both the internal storage unit and the external storage device of the electronic device 110. The memory 113 is used to store computer programs and other programs and data required by the electronic device. The memory 113 can also be used to temporarily store data that has been output or will be output.

[0025] Figure 2 A flowchart of a visual device access management method provided by an embodiment of the present application is shown. It should be understood that the size of the serial number of each step in the following embodiment does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. In this embodiment, the visual device accesses the electronic device, and the access management method adopted by the electronic device includes:

[0026] S210, obtaining the interface protocol and device information of the visual device.

[0027] The visual device is an external device of the electronic device, mainly used for inputting and collecting image data or point cloud data collected by the electronic device. The electronic device can infer the interface protocol of the visual device according to the access mode of the visual device. In some embodiments, the electronic device determines the interface protocol of the visual device according to whether the visual device is accessed through a USB interface or a Bluetooth transmission interface. The device information is necessary information to help the electronic device and the visual device to further establish a connection. In some embodiments, the electronic device can determine the type of the visual device (such as any one of an RGB camera or a depth camera) according to the data containing the device information sent by the visual device.

[0028] S220, matching the protocol specification according to the interface protocol; the protocol specification is used to translate the data electrical signal sent by the visual device into readable data.

[0029] In some embodiments, the electronic device determines the interface protocol used by the vision device according to whether the vision device is accessed through a USB interface or a Bluetooth transmission interface, and matches the protocol specification of the USB interface or the protocol specification of the Bluetooth transmission interface. The protocol specification can define the format used by the data unit, the information that the information unit should contain and the meaning, the connection mode, the timing of information sending and receiving, so as to ensure the smooth transmission of data in the communication process. For example, the USB interface and the USB protocol corresponding to the USB interface; the Ethernet interface and the Real Time Streaming Protocol (RTSP) protocol corresponding to the Ethernet interface.

[0030] S230, according to the device information, establishing a data transmission thread and a data processing thread; the data transmission thread is used for distributing readable data, and the data processing thread is used for sending the readable data distributed by the data transmission thread to an application program or a memory after processing.

[0031] In some embodiments, the electronic device can determine the type of the vision device (such as any one of an RGB camera or a depth camera) according to the device information sent by the vision device. Further, the model of the vision device can also be determined, and the mapping relationship between the device information and the model is pre-stored in the computer program or the memory, so that the hardware devices contained in the vision device can be more specifically understood. For example, the information that the vision device of model A includes an IR sensor, an RGB camera and a depth calculation engine is pre-stored in the electronic device, when the vision device of model A is accessed to the electronic device, the corresponding information can be found according to the device information sent by the vision device, and the IR data transmission sub-thread, the RGB data transmission sub-thread and the Depth data transmission sub-thread are established, which correspond to the data obtained by different sensors or processing engines respectively. It can be understood that even if the computer program or the memory of the electronic device does not pre-store any model information of the vision device, the electronic device can also access the Internet, local area network and the like, query the model or specific device information based on the vision device according to the device information, and establish the corresponding data transmission thread and data processing thread.

[0032] In some embodiments, the data transmission thread includes an RGB data transmission sub-thread, an IR data transmission sub-thread, a depth data transmission sub-thread, and / or a point cloud data transmission sub-thread. The sub-threads included in the data transmission thread can be determined according to the sensors of the visual device and the type of the processing engine. For example, when the visual device is an RGB camera, the data transmitted by the RGB camera to the electronic device only includes RGB data, and thus only an RGB data transmission sub-thread is needed to transmit the RGB data. When the visual device is an RGBD camera, the data transmitted by the RGBD camera to the electronic device can include RGB data, IR data, and depth data, and thus RGB data transmission sub-thread, IR data transmission sub-thread, and depth data transmission sub-thread are needed to transmit the RGB data, IR data, and depth data, respectively. The data transmitted by the data transmission thread can be transmitted to the data processing thread as needed, or stored in the memory of the electronic device, or transmitted to other computer programs for use.

[0033] In some embodiments, the data processing thread includes a multi-path data alignment processing sub-thread, a multi-path data synchronization processing sub-thread, and a data feature extraction sub-thread. Different data processing threads can be configured according to different data obtained from the visual device.

[0034] In some embodiments, the multi-path data alignment processing sub-thread includes: obtaining at least two frames of readable data in different formats; wherein the readable data includes RGB data and depth data; and attaching corresponding depth information to each pixel point in the RGB data according to the depth data to obtain RGBD data. The multi-path data synchronization processing sub-thread includes: obtaining data streams of at least two groups of readable data; and matching readable data with the closest timestamps in different groups according to the timestamps of the readable data in the data streams to obtain synchronized data. The data feature extraction sub-thread includes: obtaining readable data; and extracting features in the readable data by visual feature extraction or semantic feature extraction to obtain feature information.

[0035] In this embodiment, the visual device is taken as an RGBD camera as an example. In order to manage the access of the RGBD camera, the data transmission thread includes at least an RGB data transmission sub-thread and a depth data transmission sub-thread to correspond to the transmission of RGB data and depth data. Since the sensors used to collect different data are different, the specifications of the data, the frame rate of data collection, and the transmission rate can be different.

[0036] The spatial coordinate systems of the RGB data and the depth data are different, and thus the two have corresponding errors. The multi-path data alignment processing sub-thread can process the RGB data and the depth data into RGBD data. The principle of processing data by the multi-path data alignment processing sub-thread includes: converting a 2D point on the depth data into a 3D point in a world coordinate, and projecting the 3D point in the world coordinate into the RGB data. In the embodiment, only an example means for aligning the multi-path data is provided, and the embodiment of the present application should not be limited to the process of establishing the connection.

[0037] Due to different frame rates of data collection, the data frames of the multi-path data streams sampled in the same batch are not synchronized when output to the electronic device. In order to synchronize the output of the collected multi-path data, a data synchronization processing sub-thread is needed. In a specific embodiment, the data synchronization processing sub-thread includes: receiving different data frames sent by the plurality of data transmission sub-threads, adding each data frame to a corresponding queue in the queue set according to the different source sub-threads; extracting the data frame with the earliest time from each queue in the queue set that has data frames, obtaining a synchronization data packet; extracting the data frame with the earliest timestamp from the synchronization data packet as a reference data frame and outputting it to the cache, and extracting the data frame with the earliest timestamp from the remaining data frames in the synchronization data packet as a target data frame; determining whether the target data frame meets the synchronization condition according to the timestamp difference between the reference data frame and the target data frame; if yes, outputting the target data frame to the cache, and marking the data frame in the cache as a synchronization output when there is no remaining data frame in the synchronization data packet.

[0038] In order to facilitate other computer programs of the electronic device to more conveniently obtain feature information in the data transmitted by the visual device, a data feature extraction sub-thread is constructed. In some embodiments, the data feature extraction sub-thread includes a face key point recognition model, a human skeleton recognition model, an image semantic segmentation model, a pedestrian detection model, etc. After the visual device sends various data, the features in the readable data can be extracted by various neural network models in the data feature extraction sub-thread to obtain feature information.

[0039] S240, establishing a device abstraction model according to the protocol specification, the data transmission thread and the data processing thread; the device abstraction model is used to realize the data interaction between the visual device and the application program or the storage.

[0040] When a visual device first establishes a connection with an electronic device, a corresponding device abstraction model can be established for the visual device by a visual device access management method. The device abstraction model can be saved in the storage for calling when needed, or it can be established again each time the connection is re-established.

[0041] When the electronic device accesses multiple visual devices at the same time, the multiple corresponding device abstract models can be established by the visual device access management method. According to actual needs, the multiple device abstract models can also be combined into one, so as to facilitate the compatible use of multiple visual devices, or to control the operation of multiple visual devices through the same instruction.

[0042] In some embodiments, the visual device access management method further comprises: establishing a signal compiling thread according to the running environment of the electronic device; the compiling thread is used for translating the control instruction sent by the application program into a control electrical signal; and the compiling thread is added to the device abstract model.

[0043] In many cases, in addition to sending data to the electronic device, the visual device needs to receive the instruction issued by the electronic device. The visual device collects image data or point cloud data, which is not collected at all times, and needs to start a working thread according to the instruction of the electronic device. For example, in a home local area network, there is a host (electronic device) and an outdoor monitoring camera (visual device). The electronic device can send control instructions to the outdoor monitoring camera, such as camera zoom, camera pose adjustment, switching night infrared mode, etc. Due to the difference of the visual device, the same instruction of the electronic device issued to different visual devices needs to be compiled to obtain the corresponding control electrical signal, so that the visual device can correctly execute the control instruction.

[0044] In some embodiments, as shown in Figure 3 The computer program of the visual device access management method implemented by the processor when running can be packaged in a software development kit (Software Development Kit, SDK) 1131. When the program of other computer programs in the electronic device, such as image processing application programs 1132, needs to obtain the image collected by the external visual device, the API interface of the SDK can be directly called. In the prior art, when an application program needs to obtain the data of a visual device, the SDK calling mode can be used to achieve it. However, generally, a separate SDK corresponds to each external device. The visual device access management method provided in the embodiment of the present application can access multiple visual devices through the same SDK to the application program, and optimally solve the SDK calling conflict or incompatibility problem.

[0045] For the convenience of understanding, the visual device access management method provided in the present application is described by using a hierarchical structure. As shown in Figure 4 In the embodiment, the computer program of the visual device access management method can include a system layer, a communication layer, a business layer and an interface layer. The business layer includes a basic business layer and an advanced business layer.

[0046] The system layer can include code compilation files under different operating systems such as Windows, Linux, Android, MacOS, etc., so that the SDK can normally run on different operating systems. It can be understood that the corresponding code compilation files are also different for different operating systems. The signal compilation thread needs to be established according to the running environment of the electronic device; the compilation thread is used to translate the control instructions sent by the application program into control electrical signals, and different code compilation files can compile the codes of different operating systems to realize cross-platform running.

[0047] The communication layer can include one or more interface protocols, such as USB protocol and Ethernet protocol, etc. In some embodiments, the USB protocol can include one or more of USB Video Class (UVC), OPENNI, etc. The communication layer can be used to realize the communication between the software and the hardware.

[0048] The service layer can include a plurality of data transmission threads and data processing threads.

[0049] The basic service layer can include an RGB data transmission sub-thread, an IR data transmission sub-thread, a depth data transmission sub-thread, and / or a point cloud data transmission sub-thread, etc. According to the device information of the visual device, the configuration information, the number of sensors, the protocol type of the sensors, etc. of the visual device are identified. The basic service layer can obtain the data stream of the visual device. The data stream processing module can be used to process the data road. The operation instruction processing module can be used to convert the operation instruction, etc. In some embodiments, the basic service layer can include a device manager, which can be used to manage the visual device accessed.

[0050] The advanced service layer can further process the data stream of the visual device according to different operation instructions on the basis of the basic service layer. In some embodiments, the advanced service layer can include a multi-channel data alignment processing sub-thread, a multi-channel data synchronization processing sub-thread, and a data feature extraction sub-thread.

[0051] The interface layer can include an API, through which one or more functions of the SDK can be called by the application program such as a camera.

[0052] The device abstraction model can be used to access the visual device in the application program of the electronic device. In some embodiments, the device abstraction model can be used to convert the operation instructions from the application program, so as to convert the operation instructions recognizable by the application program into operation instructions recognizable by the visual device. In some embodiments, the device abstraction model can be used to process the data from the visual device, so as to convert the data into data recognizable by the application program.

[0053] The operating environment of the electronic device can include different operating systems such as Windows, Linux, Android, and MacOS in the foregoing. Of course, in actual applications, the device can also include other operating environments, and the embodiments of the present application do not limit the operating environment of the device.

[0054] In some embodiments, the visual device intervention management method matches the protocol specification, establishes a signal compilation thread, a data transmission thread, and a data processing thread, which can be specifically obtained from a configuration file set. The configuration file set includes at least one code compilation file, and each code compilation file can correspond to one operating environment. The electronic device can obtain the code compilation file corresponding to the operating environment from the configuration file set based on the operating environment of the electronic device. The configuration file set includes at least one protocol specification file, and each protocol specification file can correspond to one interface protocol. The protocol specification is used to interpret the data electrical signal and readable data sent by the visual device. The configuration file set can also include at least one data transmission logic and at least one data processing logic.

[0055] In the embodiments of the present application, the electronic device can include an application program and an SDK, and the SDK encapsulates a computer program for calling the runtime of the processor to implement the visual device access management method. The electronic device can obtain device information of a target visual device, determine a device abstraction model corresponding to the visual device based on the device information, and access the target visual device to the application program through the device abstraction model. At least one visual device can be accessed to the application program through one SDK, which improves the problems of SDK calling conflict and incompatibility when accessing the corresponding visual device to the application program through multiple SDKs.

[0056] Figure 5 A flowchart of a visual device access management method provided in the embodiments of the present application. In the present embodiment, the visual device 120 is connected to the electronic device 110 through the device interface 111, and the application program 1132 obtains the data sent by the visual device 120 to the electronic device through the SDK 1131, while the application program 1132 sends control instructions to the visual device 120 through the SDK 1131.

[0057] S510, after the visual device 120 accesses the electronic device, the processor executes the computer program of the visual device access management method to establish a device abstraction model. The protocol specification, signal compilation thread, data transmission thread, and data processing thread of the device abstraction model can be specifically obtained from a configuration file set.

[0058] S520, after the device abstract model is established, the data collected by the visual device 120 is sent to the device interface 111 in the form of an electrical signal, and the SDK 1131 obtains the data electrical signal and translates it into readable data by the device interface 111.

[0059] S521, the data transmission thread sends the readable data (such as RGB data) to the application program through the API interface.

[0060] S522, the data transmission thread sends the readable data (such as RGB data and Depth data) to the data processing thread (such as a multi-data alignment processing sub-thread), and obtains processed data (such as RGBD data).

[0061] S523, the data processing thread sends the processed data (such as RGBD data) to the application program through the API interface.

[0062] S530, the application program needs to control the visual device, so it sends a control instruction to the signal compiling thread through the API interface.

[0063] S531, the signal compiling thread translates the control instruction sent by the application program into a control electrical signal, and sends it to the visual device 120 through the device interface 111.

[0064] Figure 6 is a structural schematic diagram of a visual device access management apparatus provided by the present application. In the embodiment, the visual device access management apparatus is applied to an electronic device, which comprises:

[0065] The information acquisition module 610 is configured to acquire the interface protocol and device information of the visual device.

[0066] The protocol specification matching module 620 is configured to match a protocol specification according to the interface protocol; the protocol specification is used to translate the data electrical signal sent by the visual device into readable data.

[0067] The thread establishing module 630 is configured to establish a data transmission thread and a data processing thread according to the device information; the data transmission thread is used to distribute the readable data, and the data processing thread is used to send the readable data distributed by the data transmission thread to an application program or a storage after processing.

[0068] The model establishing module 640 is configured to establish a device abstract model according to the protocol specification, the data transmission thread and the data processing thread; the device abstract model is used to realize data interaction between the visual device and the application program or the storage.

[0069] The apparatus can further include more or fewer modules, and can implement the steps performed by the electronic device in the foregoing method embodiments, thereby achieving the effects of the method embodiments, which will not be described here.

[0070] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software function unit. In addition, the specific name of each functional unit and module is only for the convenience of mutual distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0071] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0072] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0073] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented by other means. For example, the apparatus / terminal device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed each other can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0074] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0075] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0076] The integrated module / unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of each method embodiment can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0077] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A visual device access management method, characterized by, The method is applied to an electronic device, comprising: obtaining an interface protocol and device information of a visual device; matching a protocol specification according to the interface protocol; the protocol specification is used for translating data electrical signals sent by the visual device into readable data; establishing a data transmission thread and a data processing thread according to the device information; the data transmission thread comprises an RGB data transmission sub-thread, an IR data transmission sub-thread, a Depth data transmission sub-thread and / or a point cloud data transmission sub-thread; the data processing thread comprises a multi-path data alignment processing sub-thread, a multi-path data synchronization processing sub-thread and a data feature extraction sub-thread; the data transmission thread is used for distributing the readable data, and the data processing thread is used for processing the readable data distributed by the data transmission thread and sending the processed data to an application program or a memory; establishing a device abstraction model according to the protocol specification, the data transmission thread and the data processing thread; the device abstraction model is used for realizing data interaction between the visual device and the application program or the memory.

2. The visual device access management method of claim 1, wherein, Further comprising: establishing a signal compiling thread according to a running environment of the electronic device; the compiling thread is used for translating control instructions sent by the application program into control electrical signals; adding the compiling thread to the device abstraction model.

3. The visual device access management method of claim 1, wherein, The multi-path data alignment processing sub-thread comprises: obtaining at least two frames of readable data with different formats; wherein the readable data comprises RGB data and Depth data; attaching corresponding depth information to each pixel point in the RGB data according to the Depth data to obtain RGBD data.

4. The visual device access management method of claim 1, wherein, The multi-path data synchronization processing sub-thread comprises: obtaining data streams of at least two groups of readable data; matching readable data with the closest time stamps in different groups according to time stamps of readable data in the data streams to obtain synchronization data.

5. The visual device access management method of claim 1, wherein, The data feature extraction sub-thread comprises: obtaining the readable data; extracting features in the readable data through visual feature extraction or semantic feature extraction to obtain feature information.

6. A visual device access management apparatus characterized by comprising: The device is applied to an electronic device, comprising: an information obtaining module, used for obtaining an interface protocol and device information of a visual device; a specification matching module, used for matching a protocol specification according to the interface protocol; the protocol specification is used for translating data electrical signals sent by the visual device into readable data; a thread establishing module, used for establishing a data transmission thread and a data processing thread according to the device information; the data transmission thread comprises an RGB data transmission sub-thread, an IR data transmission sub-thread, a Depth data transmission sub-thread and / or a point cloud data transmission sub-thread; the data processing thread comprises a multi-path data alignment processing sub-thread, a multi-path data synchronization processing sub-thread and a data feature extraction sub-thread; the data transmission thread is used for distributing the readable data, and the data processing thread is used for processing the readable data distributed by the data transmission thread and sending the processed data to an application program or a memory; A model establishing module is configured to establish a device abstract model according to the protocol specification, the data transmission thread and the data processing thread, and the device abstract model is configured to realize data interaction between the visual device and the application program or the memory.

7. An electronic device, comprising: The device interface is configured to establish a communication connection with the visual device. The memory is configured to store a computer program capable of running on the processor. The processor is configured to implement the steps of the method according to any one of claims 1 to 5 when executing the computer program. The computer program is configured to implement the steps of the method according to any one of claims 1 to 5 when executed by the processor.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: ​

Images