Image transmission method and device, master chip and storage medium

Abstract

The application relates to an image transmission method and device, a master control chip and a storage medium. The image transmission method comprises the following steps: receiving an acquisition scene and an image acquisition instruction sent by an upper computer; acquiring a first image acquired based on the image acquisition instruction and under the acquisition scene; writing the first image into a storage; receiving an image reading instruction sent by the upper computer, reading a second image under a target acquisition scene from the storage based on the image reading instruction; and transmitting the second image to the upper computer. The method can reduce the delay of image transmission.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to an image transmission method and device, a main control chip and a storage medium. BACKGROUND

[0002] 3D scanning is a technology of three-dimensional reconstruction through a planar diagram of structured light irradiation. The reconstructed model has quite a lot of purposes, including medicine, archaeology, reverse, industrial detection, etc. In the process of image transmission, different image collection scenes need to be faced and switched between scenes. In the traditional way, in order to ensure the stability of the collected picture when the light is switched, there is a large delay. SUMMARY

[0003] Therefore, it is necessary to provide an image transmission method, device, main control chip and storage medium capable of reducing transmission delay in view of the above technical problems.

[0004] An image transmission method is applied to a main control chip, and the method comprises the following steps:

[0005] receiving a collection scene and an image collection instruction sent by an upper computer;

[0006] acquiring a first image collected based on the image collection instruction and in the collection scene;

[0007] writing the first image into a memory;

[0008] receiving an image reading instruction sent by the upper computer, and reading a second image in a target collection scene from the memory based on the image reading instruction;

[0009] transmitting the second image to the upper computer.

[0010] An image transmission device comprises the following:

[0011] an upper computer data interface module configured to receive a collection scene and an image collection instruction sent by an upper computer;

[0012] an image analysis module configured to acquire a first image collected based on the image collection instruction and in the collection scene;

[0013] a read-write control module configured to write the first image into a memory;

[0014] the upper computer data interface module is configured to receive an image reading instruction sent by the upper computer;

[0015] the read-write control module is configured to read a second image in a target collection scene from the memory based on the image reading instruction;

[0016] The host computer data interface module is configured to transmit the second image to the host computer.

[0017] A master control chip for implementing the steps of the image transmission method.

[0018] A computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the image transmission method.

[0019] The image transmission method, device, master control chip and storage medium described above receive an acquisition scene and an image acquisition instruction sent by a host computer, acquire an image under the acquisition scene based on the image acquisition instruction, obtain a first image, write the first image into a storage, and then read a second image under a target acquisition scene from the storage based on an image reading instruction, and transmit the second image to the host computer. That is, in order to adapt to various acquisition scenes and flexibly regulate and control the acquisition scene to meet the demand, the first image is stored in the storage outside the host computer, which can be selected by the host computer, and part of the second image can be read out in the image transmission process. Compared with the way of completely reading out and reading the image in the host computer, the image reading rate is improved, and the switching delay of the scene image and the image transmission delay are reduced. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 An application environment diagram of the image transmission method in one embodiment;

[0021] Figure 2 A flowchart of the image transmission method in one embodiment;

[0022] Figure 3 A transmission diagram of read-write multiplexing of the storage in one embodiment;

[0023] Figure 4 A structure diagram of the image transmission in one embodiment;

[0024] Figure 5 A diagram of a DDR control state machine in one embodiment;

[0025] Figure 6 A structure diagram of the image transmission in another embodiment;

[0026] Figure 7 A flowchart of the image transmission method in one embodiment;

[0027] Figure 8 A structure block diagram of the image transmission device in one embodiment. DETAILED DESCRIPTION

[0028] It should be understood that the specific embodiments described herein are merely illustrative of the present application and do not limit the present application.

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort fall within the scope of the present application.

[0030] It should be explained that all the direction indications (such as up, down, left, right, front, back, and the like) in the embodiments of the present application are only used to explain the relative position relationship, movement condition, and the like between components in a certain specific posture (as shown in the drawings). If the specific posture changes, the direction indications also change accordingly. The connection can be direct connection or indirect connection.

[0031] In addition, the descriptions such as “first”, “second” and the like in the present application are only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features defined as “first”, “second” can explicitly or implicitly include at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it. When the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the protection scope of the present application.

[0032] The terms “first”, “second”, and the like used in the present application can be used herein to describe various data, but these data are not limited by these terms. These terms are only used to distinguish the first data from the other data. For example, without departing from the scope of the present application, the first image can be called the second image, and similarly, the second image can be called the first image. The first image and the second image are both images, but they are not the same image. The first image refers to an acquired image, and the second image refers to a read image, which represent different meanings, but can be the same image.

[0033] It can be understood that “connection” in the following embodiments means “electrical connection”, “communication connection”, and the like if the connected circuits, modules, units, and the like have the transmission of electrical signals or data to each other.

[0034] The image transmission method provided by the present application can be applied to the application environment such as Figure 1 The application environment of the image transmission method in an embodiment is shown in the figure. Figure 1 The application environment of the image transmission method in an embodiment is shown in the figure. Figure 1The system includes an image acquisition device 110, a master chip 120, and a host computer 130. The image acquisition device 110 can be a camera, an image sensor such as a CCD, a CMOS sensor, etc. The image acquisition device 110 can be two, used to acquire images at different positions to achieve binocular image acquisition. The master chip 120 can be an FPGA (Field Programmable Gate Array), an ARM (Advanced RISC Machine) chip containing an image interface, etc. The master chip 120 can also be located in a computer device. The memory can be built-in in the master chip 120 or external to the master chip 120. The host computer 130 can be a computer device. The computer device can be, but is not limited to, various personal computers, notebook computers, smartphones, tablet computers, and portable wearable devices. The host computer 130 sends an image acquisition instruction containing different acquisition scenes, and the master chip 120 sends the image acquisition instruction to the image acquisition device 110. The image acquisition device 110 acquires images in the acquisition scene based on the image acquisition instruction to obtain a first image. The master chip 120 writes the first image into the memory. The host computer 130 sends an image reading instruction, and the master chip 120 reads a second image in the corresponding scene from the memory based on the image reading instruction and transmits the second image to the host computer. The embodiments of the present application take the example of the image acquisition device 110 being external to the master chip 120 for specific description.

[0035] In one embodiment, as shown in FIG. 8, it is a flowchart of an image transmission method in one embodiment. The method is applied in a master chip, wherein: Figure 2

[0036] Step 202, receiving an acquisition scene and an image acquisition instruction sent by the host computer.

[0037] Specifically, the acquisition scene can be a light control scene. The acquisition scene can be included in the image acquisition instruction or sent separately. By adding the acquisition scene to the image acquisition instruction, the scene switching delay can be further reduced. The image acquisition instruction can include the acquisition scene and an image acquisition trigger, and can also include a specified image storage address. The number of acquisition scenes in the image acquisition instruction is not limited.

[0038] Step 204, obtaining a first image acquired based on the image acquisition instruction and in the acquisition scene.

[0039] The first image refers to an image acquired by the image acquisition device based on the image acquisition instruction in the acquisition scene.

[0040] ​Specifically, the master chip parses the image acquisition instruction to obtain an acquisition scene, controls the scene implementation module to configure the acquisition scene, and acquires an image by the image acquisition device in the acquisition scene to obtain a first image. For example, the acquisition scene is a single-line laser control scene, the scene implementation module is switched to single-line laser, and the image acquisition device acquires an image in the scene and transmits the image to the master chip. The master chip obtains the first image acquired based on the image acquisition instruction in the acquisition scene.

[0041] In this embodiment, when the number of acquisition scenes is more than one, the master chip can control the scene implementation module to switch scenes according to the order of the acquisition scenes in the image acquisition instruction.

[0042] Step 206: write the first image into the memory.

[0043] The memory can be built-in or external to the master chip. The memory can be a RAM (Random Access Memory) or a ROM (Read-Only Memory). The memory is not the memory in the host computer.

[0044] Specifically, the master chip writes the first image into the corresponding image storage address in the memory based on the image storage address parsed from the image acquisition instruction.

[0045] Step 208: receive an image reading instruction sent by the host computer, and read a second image in a target acquisition scene from the memory based on the image reading instruction.

[0046] The image reading instruction includes an image reading address, an image reading data amount, etc. The second image refers to the image read by the host computer. The target acquisition scene refers to the acquisition scene specified in the image reading instruction. The target acquisition scene can also be represented by the image reading address.

[0047] Specifically, the master chip parses the image reading instruction to obtain an image reading address, and reads the second image in the corresponding scene from the memory based on the image reading address. For example, the master chip acquires 10 first images in the first acquisition scene and 10 first images in the second acquisition scene, and writes them into the memory in sequence. The host computer sends an image reading instruction, which specifies the image reading address of the 15th image. The master chip reads the second image in the second acquisition scene based on the image reading address of the 15th image.

[0048] Step 210: transmit the second image to the host computer.

[0049] Specifically, the master chip transmits the second image through the data interface of the host computer.

[0050] In this embodiment, the acquisition scene and the image acquisition instruction sent by the host computer are received, the first image is acquired based on the image acquisition instruction under the acquisition scene, the first image is written into the memory, and then the second image under the target acquisition scene is read from the memory based on the image reading instruction, and the second image is transmitted to the host computer. That is, in order to adapt to various acquisition scenes and flexibly regulate and control the acquisition scene to meet the needs, the first image is stored in the memory outside the host computer, which can be selected by the host computer, and part of the second image can be read out during image transmission. Compared with the way of completely reading out and reading out the image in the host computer, the image reading rate is improved, and the switching delay of the scene image and the image transmission delay are reduced.

[0051] In one embodiment, writing the first image into the memory includes:

[0052] The first image transmitted through the image transmission protocol is written into the memory at a writing speed supported by the memory; the writing speed is greater than the transmission speed of the image transmission protocol;

[0053] The image reading instruction sent by the host computer is received, and the second image under the target acquisition scene is read from the memory based on the image reading instruction, including:

[0054] The image continuous reading instruction sent by the host computer is received, and the second image under the acquisition scene written in the last time period is read from the memory at a reading speed supported by the memory;

[0055] Return to the step of writing the first image transmitted through the image transmission protocol into the memory at a writing speed supported by the memory, until the reading stop instruction of the host computer is detected, and the reading of the second image under the target acquisition scene from the memory is stopped.

[0056] The transmission speed of the image transmission protocol can be the transmission speed of MIPI (Mobile Industry Processor Interface). The writing speed supported by the memory can be greater than twice the transmission speed of the image transmission protocol. Similarly, the reading speed supported by the memory is greater than the transmission speed through the host computer data interface. The memory is not supported to be read at the same time when writing. Taking the memory as an example, the full load speed of DDR is greater than the MIPI transmission speed + the host computer data interface transmission speed.

[0057] The image continuous reading instruction means that the host computer needs to read one frame of image when the image acquisition device acquires one frame of image. The reading stop instruction can be a stop symbol in the image stop instruction, or a stop instruction re-sent by the host computer.

[0058] Specifically, after receiving the image acquisition instruction, the master chip writes the first image transmitted through the image transmission protocol into the memory at a writing speed supported by the memory. The image continuous reading instruction sent by the upper computer is received. The second image under the acquisition scene written in the last time period is the second image under the target acquisition scene. The second image under the acquisition scene written in the last time period is read from the memory at a reading speed supported by the memory. After reading, the master chip returns to execute the step of writing the first image transmitted through the image transmission protocol into the memory at a writing speed supported by the memory in real time until the reading stop instruction of the upper computer is detected, and the reading of the second image under the target acquisition scene from the storage area is stopped. Then, the read second image is transmitted to the upper computer through the upper computer data interface. That is, the reading process of the second image is inserted in the writing process of the first image.

[0059] As shown in Figure 3 , it is a transmission schematic diagram of memory read-write multiplexing in an embodiment. The memory can be a memory that does not support simultaneous reading and writing. In this embodiment, the memory is taken as an example for description. The MIPI transmission data time before improvement refers to the time spent in transmitting the image through the MIPI. One block can represent one frame of the first image. The USB receiving data time before improvement refers to the time spent in receiving the second image through the USB module. Before improvement, the MIPI channel transmits the first image, and the DDR writes the first image. Since the DDR writing speed is much greater than the MIPI transmission speed, the writing is idle for a period of time. Similarly, the DDR reading speed is much greater than the USB receiving data speed, so the DDR is idle for a period of time after reading for a period of time. Therefore, the DDR working time t1 before improvement is the MIPI transmission data time + the USB receiving data time.

[0060] After improvement, the MIPI transmission data time does not change, but due to the read-write multiplexing mode of the DDR, the writing is performed first, and then the reading is performed, and the process is repeated. Therefore, the DDR working time after improvement is the longer one of the MIPI transmission data time and the USB receiving data time, which is the USB receiving data time after improvement in Figure 3 . Since there are actually hundreds of times of reading and writing alternation in the process of transmitting each frame of image, the first writing time in the t3 time can be ignored, that is, t2=t3. It can be known from Figure 3 that in the memory, the time consumption of the traditional read-then-write mode t1 is much longer than that of the read-write multiplexing mode t3. After improvement, the DDR working state is compared with the DDR working state before improvement, and a large amount of idle time is reduced, and a large amount of transmission time consumption is reduced.

[0061] By alternately occupying the interface of DDR3 for reading and writing, the highest image transmission resolution can reach 1920*1200*8*2 bits, and the transmission frame rate is 58 fps.

[0062] In the embodiment, the first image transmitted through the image transmission protocol is written into the memory at a writing speed supported by the storage area, and after the image continuous reading instruction sent by the host computer, the second image under the collected scene written at the previous time period is read from the memory at a reading speed supported by the memory, and the step of writing is returned to execution, that is, the reading and writing of the memory are multiplexed by alternately reading and writing, so that the image transmission meets the requirement of high frame rate, the image transmission speed is improved, and the transmission time is reduced.

[0063] In one embodiment, the first image transmitted through the image transmission protocol is written into the memory at a writing speed supported by the memory, including:

[0064] The first image transmitted through the image transmission protocol is stored in the first-in-first-out module, and the first image is read out from the first-in-first-out module and written into the memory at a writing speed supported by the memory.

[0065] The second image is transmitted to the host computer, including:

[0066] The second image is transmitted to the host computer through the first-in-first-out module.

[0067] The first-in-first-out module refers to a FIFO (First Input First Output) module, which can realize the problem of cross-clock synchronization domain.

[0068] Specifically, the first-in-first-out module is connected between the image acquisition device and the memory, and the first-in-first-out module is also connected between the memory and the host computer. The first image transmitted through the image transmission protocol is stored in the first-in-first-out module by the master control chip, and the first image is read out from the first-in-first-out module and written into the memory at a writing speed supported by the memory. The master control chip receives the image continuous reading instruction sent by the host computer, reads the second image under the collected scene written at the previous time period from the memory at a reading speed supported by the memory.

[0069] In the embodiment, the first image transmitted through the image transmission protocol is stored in the first-in-first-out module, and the first image is read out from the first-in-first-out module and written into the memory at a writing speed supported by the memory, and the target image is transmitted to the host computer through the first-in-first-out module, which can maintain clock synchronization through the first-in-first-out module, and when the channel is blocked when transmitting the second image to the host computer, image data overflow can also be avoided.

[0070] In one embodiment, the image transmission method further comprises: when the second image is lost, generating a lost frame filling frame based on preset data agreed with the host computer;

[0071] transmitting the lost frame filling frame to the host computer; the lost frame filling frame is used to instruct the host computer to send a first reset instruction for resetting the image acquisition device and the master control chip;

[0072] receiving the first reset instruction and resetting the image acquisition device and the master control chip based on the first reset instruction.

[0073] The preset data agreed with the host computer can be specifically image data that does not appear in normal image shooting. For example, the preset data can be black and white stripe alternating data, pure color image data, and television snowflake screen, etc. The lost frame filling frame is an image, and can also be a stripe alternating image frame, a pure color image frame, a television snowflake image frame, etc.

[0074] Specifically, the master control chip can count or time, and when the second image is not read within a preset time length, it is determined that the second image is lost. When the second image is read, it is read frame by frame, and if there is a lost frame, the master control chip directly transmits to the host computer through the USB module, and the USB module stores data in blocks. Each time the host computer can only read data according to a multiple of the minimum block size, and in the case of a lost frame, the minimum block is not satisfied. At this time, the USB module cannot work when the host computer reads, resulting in a long image transmission time.

[0075] When the second image is lost, a lost frame filling frame is generated based on preset data agreed with the host computer. The master control chip transmits the lost frame filling frame to the host computer, and the host computer sends a first reset instruction for resetting the image acquisition device and the master control chip. The master control chip receives the first reset instruction, restarts based on the first reset instruction, and controls the image acquisition device to reset. Alternatively, the master control chip receives the first reset instruction and sends the first reset instruction to the image acquisition device for resetting.

[0076] In this embodiment, when a frame is lost, a lost frame filling frame is generated based on preset data agreed with the host computer, and the lost frame filling frame is transmitted to the host computer, which avoids the collapse of the data transmission interface of the host computer, resulting in a long image transmission time. In the case of a lost frame failure, it can be that the image acquisition device has a bug, or it can be that the master control chip has a bug. By receiving the first reset instruction sent by the host computer, the image acquisition device and the master control chip are reset, which can solve the failure and improve the image transmission efficiency.

[0077] In one embodiment, the image transmission method further comprises: when the second image is lost, generating a lost frame filling frame based on preset data agreed with the host computer;

[0078] transmitting a lost-line completion frame to the host computer; the lost-line completion frame is used to instruct the host computer to send a second reset instruction for resetting the master chip;

[0079] receiving the second reset instruction and resetting the master chip based on the second reset instruction.

[0080] The lost line refers to at least one part being lost. The lost-line completion frame refers to a frame composed of the preset data agreed with the host computer and the image data that is not lost. That is, the lost data is completed by the preset data agreed with the host computer.

[0081] Specifically, the master chip can count or time, and when a complete second image is not read within a preset time length or the time length for receiving the second image exceeds the preset time length, it is determined that the second image is lost. When reading the second image, the second image is read frame by frame, and if there is a lost line, the master chip directly transmits the lost line to the host computer through the USB module. The USB module stores data in blocks, and each time the host computer can only read data according to a multiple of the minimum block size. In the case of lost frames, the minimum block is not satisfied, and at this time, the USB module fails to work when the host computer reads, resulting in a long image transmission time.

[0082] When the second image is lost, a lost-line completion frame is generated based on the read second image and the preset data agreed with the host computer. The master chip transmits the lost-line completion frame to the host computer, and the host computer sends a second reset instruction for resetting the master chip. The master chip receives the second reset instruction and restarts based on the second reset instruction.

[0083] In this embodiment, when the line is lost, a lost-line completion frame is generated based on the preset data agreed with the host computer, and the lost-line completion frame is transmitted to the host computer, which avoids the collapse of the data transmission interface of the host computer, resulting in a long image transmission time. In the case of a lost line fault, there is a high probability that the master chip has a bug. By receiving the second reset instruction sent by the host computer, the master chip is reset, which can solve the fault and improve the image transmission efficiency.

[0084] In one embodiment, writing the first image into the memory comprises:

[0085] Converting the bit width of the first image transmitted through the image transmission protocol to obtain an image with a bit width adapted to the memory, and writing the first image with the bit width adapted to the memory into the memory;

[0086] transmitting the second image to the host computer, comprising:

[0087] Converting the bit width of the second image read from the memory to obtain a second image with a bit width adapted to the data interface of the host computer;

[0088] Transmitting the second image adapted to the data interface of the host computer to the host computer through the data interface of the host computer.

[0089] The host computer data interface refers to a data interface connected to the host computer, such as a USB module, a network interface, a thunderbolt interface, and the like.

[0090] Specifically, the master chip performs bit width conversion on the first image transmitted through the image transmission protocol to obtain an image with bit width adapted to the memory, and writes the first image with bit width adapted to the memory into the memory. The master chip performs bit width conversion on the second image read from the memory to obtain a second image with bit width adapted to the host computer data interface, and transmits the second image adapted to the host computer data interface through the host computer data interface. For example, the first image is 32-bit MIPI data, and the memory stores 128-bit data, and the image transmitted by the USB module is 32-bit data. Then the master chip performs bit width conversion on the 32-bit MIPI first image to obtain 128-bit data and stores it into the memory; performs bit width conversion on the 128-bit second image to obtain a 32-bit second image, and transmits the 32-bit second image through the USB module interface.

[0091] In this embodiment, the first image transmitted through the image transmission protocol is converted in bit width to obtain an image with bit width adapted to the memory, the first image with bit width adapted to the memory is written into the memory, the second image read from the memory is converted in bit width to obtain a second image with bit width adapted to the host computer data, and the second image adapted to the host computer data interface is transmitted through the host computer data interface, so that a large amount of images originally stored in the host computer can be stored in the memory of the peripheral device, and the image transmission efficiency is improved.

[0092] In one embodiment, the collection scene includes at least one of a color control scene of light, a brightness control scene of light, a laser type control scene, and a light supplement lamp control scene.

[0093] Specifically, the color control scene of light refers to controlling the color change of visible light, such as presenting red, orange, yellow, green, cyan, blue, purple, and the like.

[0094] The brightness control scene of light refers to controlling the brightness change of light. The laser type control scene refers to controlling the laser irradiated on the object to be single-line laser, multi-line laser, and the like. The light supplement lamp control scene refers to light control for supplementing light to the object.

[0095] In this embodiment, through the color control of light, the brightness control of light, the laser type control, and the light supplement lamp control, images under different collection scenes can be tested, calibrated, and processed, and the like, to meet various requirements; and by improving the image transmission efficiency, the delay caused by switching between collection scenes can be reduced.

[0096] In one embodiment, the 3D scanning is a technology of reconstructing three-dimensionally through a planar diagram of structured light irradiation. The reconstructed model has quite many uses, including medicine, archaeology, reverse, industrial detection, etc. With the development of 3D scanning technology, the required accuracy of 3D scanning is gradually improved, so the requirements for the resolution and frame rate of the image are also improved, and there are different requirements for the image acquisition situation when calibrating and scanning the 3D scanning.

[0097] The traditional image acquisition control module and the scene control module are often separated, the instruction is sent to control the scene first, then the image is transmitted, when the acquisition scene needs to be switched, the image transmission needs to be stopped, then the light is switched, and then the image is transmitted. In order to ensure the stability of the acquired picture when the structured light needs to be switched, there is a large transmission delay and acquisition scene stability delay.

[0098] The purpose of the embodiment of the application is to improve the resolution and frame rate, and to enable the upper computer to flexibly control various image acquisition scenes to meet the current needs, while reducing the delay in switching between scenes.

[0099] The purpose of the application is to provide a kind of based on FPGA and can be controlled by upper computer, high resolution high frame rate binocular image acquisition system implementation scheme. With image acquisition device as MIPI CMOS sensor, main control chip is FPGA, memory is DDR3, main control chip peripheral laser is used to realize laser type control scene, LED fill light is used to realize fill light control, and upper computer data interface is USB3.0 chip as an example for description.

[0100] The application adopts MIPI CMOS sensor, FPGA, DDR3, laser, LED fill light and USB3.0 chip. Among them:

[0101] The MIPI CMOS sensor is used to collect image information, and can be configured by the FPGA through the issued instruction to expose, flip, etc.

[0102] The USB3.0 module is responsible for the issuance of upper computer instructions and the uploading of image data, and the types of instructions include image acquisition instructions, image reading instructions, CMOS sensor configuration control, laser and LED brightness duty cycle control.

[0103] DDR3 is used to store the image data after being unpacked from the MIPI CMOS and the instructions issued by the upper computer, to transmit a certain amount of data specified by the instructions to the upper computer.

[0104] FPGA as the central control, responsible for receiving and unpacking MIPI image data, USB instructions, control DDR read-write interface, generate the agreed USB package data transmission to the USB chip, configuration CMOS and configuration laser voltage function implementation.

[0105] The laser is to provide the required structured light for 3D scanning.

[0106] By reading and writing alternately occupying the interface of DDR3, the highest image transmission resolution can reach 1920 * 1200 * 8 * 2bits, and the transmission frame rate is 58fps.

[0107] The embodiment of the application is a high-resolution high-frame-rate binocular image acquisition system based on FPGA and capable of being controlled by a host computer. The overall framework diagram of the whole scheme is shown in Figure 4 As shown in Figure 4 is a structural diagram of image transmission in an embodiment. The FPGA includes a USB control module, a DDR read-write control module, a MIPI data unpacking module, a USB to SPI module, a PWM scene control module and a USB control logic. The image transmission structure also includes a frame loss and frame filling module, a bit width conversion module and a key control module.

[0108] The USB control logic includes a USB to IIC Master module. The USB to IIC Master module is controlled by the FPGA to send the IIC protocol, and then provides a register to the host computer. The host computer can control the IIC address, register address, address length, register value and value length of the IIC transmission by issuing corresponding USB instructions. The host computer can also read the data in the received value register through the USB instruction. The function of the IIC Master is to control the register of the MIPI CMOS sensor, which can configure exposure, resolution, frame rate, working mode, etc.

[0109] The MIPI data unpacking module is mainly used for analyzing MIPI image data. Since it is binocular image transmission, the MIPI unpacking module contains two in total.

[0110] DDR read-write control module includes bit width conversion module, 128-bit FIFO, mig ip (mig: Memory Interface Generators) and app interface control module. Because MIPI image data is 32-bit, and the data interface of DDR3 ip is 128-bit, a bit width conversion module is inserted in the middle. The bit width conversion module is mainly used for converting 32-bit MIPI data to 128-bit DDR and converting 128-bit DDR read-out data to 32-bit FX3 interface of USB. Because the system clock of mig ip and the clock of axi_stream protocol are not synchronized, an asynchronous FIFO is added between them for data transmission. The mig ip is an ip used for controlling DDR3 read-write, and the chip specification parameters, data bit width, system clock, pin allocation, etc. of DDR3 are set through a configuration page. After the setting is completed, the ip provides an app interface for users. The control of read-write is controlled by app_cmd signal, so it is indicated that read-write cannot be performed simultaneously. However, because the full load rate of DDR > MIPI two-channel rate + USB transmission rate, read-write multiplexing of MIPI channel and USB transmission can be realized by alternating read-write interface and adding a large-capacity FIFO. The specific scheme is as follows: (1) MIPI image data is stored in a large-capacity FIFO, and program_full and program_empty provided by the FIFO are used to control whether the current channel image data occupies or exits the app interface of the mig ip. The large capacity of the FIFO is used to ensure that when other channels occupy the app interface, even in the worst case, the image data will not overflow. (2) The working mode of MIPI cmos sensor is trigger mode, that is, one frame of image is sent after receiving a trigger signal once. In order to ensure that the large-capacity FIFO is read empty every time the trigger is triggered, the number of data count control is also added in the control of the app interface. (3) The data read out by DDR is also first entered into a FIFO, and the full and empty of the FIFO are used to control the occupation and exit of the app interface. After the data is read out, the bit width is converted from 128-bit to 32-bit to adapt to the interface provided by the USB chip. The state machine control flow of the whole module is shown in Figure 5 . Figure 5 The DDR control state machine in an embodiment is shown in the figure. It includes idle state, MIPI channel 1 write state, that is, WRITE_DDR1, MIPI channel 2 write state, that is, WRITE_DDR2, arbitration state ARBITRATION, read state READ_DDR, frame filling state FRAME_FIX1 and line filling state FRAME_FIX2.

[0111] The USB control module mainly includes unpacking, packing and interface control. Since the FPGA is connected with the USB chip of Cypress, the chip can provide an FX3 interface similar to the FIFO interface to the FPGA. The interface control is to control the read and write states of the FX3 interface through a state machine. When in the read state, the unpacking module starts to work, analyzes and unpacks the instructions according to the agreed instructions of the host computer, and assigns the obtained data to the corresponding instruction register. These instruction registers can control the laser, LED brightness, transmission of the trigger signal of the CMOS sensor, color control of the RGB lamp, the first address of the DDR write and read, the data quantity of the DDR read, etc. When in the write state, the packing module starts to work. The returned frame data of the packing is divided into two categories. One is the normal frame return, which is used to tell the host computer whether the FPGA receives the last issued USB instruction normally and the current working state information of each module. The other is the image frame return, including the frame header, image data and frame tail check. The frame header also includes part of the working state information of the module.

[0112] The principles of the USB-to-SPI module and the USB-to-IIC module are consistent, both of which are to control the output of the relevant protocol waveform through the USB instruction. The main function of the SPI is to communicate with the chip for controlling the power supply voltage of the laser, and to control the brightness of the laser by changing the power supply voltage of the laser through the SPI.

[0113] The frame loss and frame supplement module is used to send the specific data agreed with the host computer to the host computer when the FPGA sends the trigger signal without receiving the image information due to the problems of the csi2_rx ip or the CMOS sensor, so as to let the host computer send the corresponding logical reset signal or reconfiguration signal. The specific implementation scheme is to add a timer in the process of sending the image acquisition signal to receiving the first image data, and to start supplementing the false image data frame once the timer is timed out.

[0114] The key control module records the number of times of pressing the key, and transmits the recorded key action times to the host computer in the form of packing when the host computer queries each time, and clears the record after the transmission is completed.

[0115] Figure 6The structure diagram of image transmission in another embodiment. The host computer sends data through the FX3 interface of USB3.0, and sends operation instructions through the USB3.0 control module. The operation instructions can include the acquisition scene and the image acquisition instruction. It can be understood that the image acquisition instruction can include the acquisition scene; then the exposure time when the image acquisition device acquires the first image can be used as the stable time of the next scene switching (at this time, image transmission is also being performed). Therefore, by combining the acquisition scene and the image acquisition instruction, the scene switching delay and the host computer communication delay can be reduced. The host computer sends operation instructions to the IIC host, which converts the operation instructions into IIC sentences and transmits them to the image acquisition device. The image acquisition device acquires the first image under the acquisition scene and transmits the first image to the MIPI data unpacking module in the form of MIPI data. After the first image is converted from 32 bits to 128 bits, it is transmitted to the FIFO, and the DDR3 read-write control module writes the first image to the DDR3 through the Command interface. When receiving the image reading instruction sent by the host computer, the DDR3 read-write control module reads the second image corresponding to the target acquisition scene from the DDR3 through the Command interface and transmits it to the FIFO. The data is converted from 128 bits to 32 bits and uploaded to the USB3.0 control module and the host computer.

[0116] Figure 7 The flowchart of the image transmission method in one embodiment is shown in the figure below. The specific operation is as follows:

[0117] 1. Confirm whether the pre-requisite configuration is completed. The pre-configuration includes USB3.0 chip firmware burning and MIPI cmos sensor configuration. If not, perform the above pre-configuration.

[0118] 2. After completing the pre-configuration, confirm the required light and laser brightness, and configure them through the host computer. (That is, the host computer sends the acquisition scene to the FPGA.)

[0119] 3. After completing the above configuration, if image acquisition is selected, perform 4; if image reading is selected, perform 10.

[0120] 4. The host computer sends an image acquisition instruction (which is used to trigger the cmos sensor to work and indicate the starting address of the DDR image data storage).

[0121] 5. Detect whether the received image data has frame loss. After sending the image acquisition signal to the cmos sensor, timing is performed. If timeout occurs, it is determined that frame loss has occurred, and step 8 is performed; if the timer does not time out, step 6 is performed.

[0122] 6、Detect whether the received image data is received completely, whether the line loss occurs. Also, there is a timer to time the time from receiving the image data to receiving the completion. If it is timed out, it is determined as line loss, and 9 is executed; if it is not timed out, 7 is executed.

[0123] 7、Control the DDR to store the image data into the DDR address specified by the host computer. At this time, the image acquisition process ends.

[0124] 8、After determining the frame loss, the frame filling module starts to replace the MIPI data unpacking module to provide the special data agreed with the host computer to the DDR control module (the host computer will send the instruction to reset the fpga and cmos sensor after seeing the data). Then 7 is executed.

[0125] 9、After determining the line loss, the frame filling module starts to replace the MIPI data unpacking module to provide the special data agreed with the host computer to the DDR control module (the host computer will send the instruction to reset the fpga after seeing the data). Then 7 is executed.

[0126] 10、The host computer sends the image reading instruction (the instruction is used to trigger the DDR to output the image data and indicate the first address and data amount of the DDR output data).

[0127] 11、The image data is written into the USB chip buffer through the fx3 interface after bit width conversion and FIFO.

[0128] 12、After the USB chip obtains the data, the host computer reads it. At this time, the image data reading process ends.

[0129] In the embodiment, the USB and the FPGA can provide various switching requirements to ensure flexibility, and the function of directly implementing the USB instruction through the FPGA can reduce the switching delay. In order to enable the DDR SDRAM to meet the requirements of high frame rate and high resolution, read-write multiplexing of the front and rear ends is added. In this way, the overall scheme can improve the integration of the system and reduce the cost of the device, while also ensuring the accuracy of 3D scanning to a certain extent. At present, it can smoothly switch among various structured light requirements and ensure the stability of the structured light. At the same time, since the DDR interface is opened to the host computer, the host computer can flexibly acquire and read images. In addition, the keys on the device can meet most of the functional requirements in daily use, so that the UI interface of the host computer does not need to be clicked frequently. At present, the accuracy of the structured light can be maintained within 0.1mm to 2mm.

[0130] In one embodiment, an image transmission method applied to a master control chip, comprising:

[0131] Step (a1), receiving the collection scene and the image collection instruction sent by the host computer.

[0132] Step (a2), collecting the image under the collection scene based on the image collection instruction to obtain a first image.

[0133] Step (a3), performing bit width conversion on the first image transmitted through the image transmission protocol to obtain a first image with bit width adapted to the memory, and storing the first image with bit width adapted to the memory in the first-in-first-out module at a writing speed supported by the memory.

[0134] Step (a4), writing the first image with bit width adapted to the memory into the memory. The writing speed is greater than the transmission speed of the image transmission protocol.

[0135] Step (a5), receiving the image continuous reading instruction sent by the host computer, and reading the second image under the collection scene written in the previous time period from the memory at a reading speed supported by the memory.

[0136] Step (a6), returning to the step of writing the first image into the memory at a writing speed supported by the memory until the reading stop instruction sent by the host computer is detected, and stopping reading the second image under the target collection scene from the memory.

[0137] Step (a7), performing bit width conversion on the second image read from the memory to obtain a second image with bit width adapted to the data interface of the host computer.

[0138] Step (a8), sequentially transmitting the second image adapted to the data interface of the host computer through the first-in-first-out module and the data interface of the host computer.

[0139] Step (a9), when the second image is lost frame, generating a lost frame filling frame based on preset data agreed with the host computer.

[0140] Step (a10), transmitting the lost frame filling frame to the host computer. The lost frame filling frame is used to instruct the host computer to send a first reset instruction for resetting the image collection device and the master control chip.

[0141] Step (a11), receiving the first reset instruction, and resetting the image collection device and the master control chip based on the first reset instruction.

[0142] Step (a12), when the second image is lost line, generating a lost line filling frame based on the read second image and preset data agreed with the host computer.

[0143] Step (a13), transmitting the lost line filling frame to the host computer. The lost line filling frame is used to instruct the host computer to send a second reset instruction for resetting the master control chip.

[0144] Step (a14), receiving a second reset instruction, and resetting the master chip based on the second reset instruction.

[0145] In this embodiment, the acquisition scene and the image acquisition instruction sent by the host computer are received, the first image is acquired based on the image acquisition instruction under the acquisition scene, the first image is written into the storage, and then the second image under the target acquisition scene is read from the storage based on the image reading instruction, and the second image is transmitted to the host computer. That is, in order to adapt to various acquisition scenes and flexibly regulate and control the acquisition scene to meet the demand, the first image is stored in the storage outside the host computer, which can be selected by the host computer. In the image transmission process, part of the second image can be read out. Compared with the complete reading out mode, the image transmission rate is improved, the scene switching delay and the image transmission delay are reduced. In the case of frame loss or line loss, the corresponding device is reset in time to solve the fault and improve the image transmission efficiency.

[0146] It should be understood that, although the above Figure 2 and Figure 7 The steps in the flowchart are displayed in sequence according to the arrow, and the steps (a1) to (a14) are displayed in sequence according to the number. However, these steps are not necessarily executed in sequence according to the arrow or the number. Unless otherwise stated in this article, the execution of these steps has no strict order limitation, and these steps can be executed in other order. Moreover, Figure 2 and Figure 7 At least part of the steps in and

[0147] These steps or stages do not necessarily execute at the same time, but can execute at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps. Figure 8 Figure 8 In one embodiment, as shown in , it is a structure block diagram of the image transmission device in one embodiment.

[0148] An image transmission device is provided, which can be a part of a computer device using software modules or hardware modules, or a combination of the two. The device specifically includes: a host computer data interface module 802, an image analysis module 804 and a read-write control module 806, wherein:

[0149] The host computer data interface module 802 is used to receive the acquisition scene and the image acquisition instruction sent by the host computer;

[0150] The read-write control module 806 is configured to write the first image into the memory.

[0151] The host computer data interface module 802 is configured to receive an image reading instruction sent by the host computer.

[0152] The read-write control module 806 is configured to read a second image under a target acquisition scene from the memory based on the image reading instruction.

[0153] The host computer data interface module 802 is configured to transmit the second image to the host computer.

[0154] In this embodiment, the acquisition scene and the image acquisition instruction sent by the host computer are received, the image under the acquisition scene is acquired based on the image acquisition instruction to obtain the first image, the first image is written into the memory, and then the second image under the target acquisition scene is read from the memory based on the image reading instruction, and the second image is transmitted to the host computer. That is, in order to adapt to various acquisition scenes and flexibly control the acquisition scene to meet the demand, the first image is stored in the memory outside the host computer, which can be selected by the host computer, and part of the second image can be read out in the image transmission process. Compared with the mode of completely reading out and reading out the image in the host computer, the image reading rate is improved, and the switching delay of the scene image is reduced.

[0155] In one embodiment, the read-write control module 806 is configured to write the first image transmitted through the image transmission protocol into the memory at a writing speed supported by the memory; and the writing speed is greater than the transmission speed of the image transmission protocol.

[0156] The host computer data interface module 802 is configured to receive an image continuous reading instruction sent by the host computer.

[0157] The read-write control module 806 is configured to read the second image under the acquisition scene written in the last time period from the memory at a reading speed supported by the memory; and until a reading stop instruction of the host computer is detected, the reading of the second image under the target acquisition scene from the memory is stopped.

[0158] In this embodiment, the first image transmitted through the image transmission protocol is written into the memory at a writing speed supported by the memory, and after the image continuous reading instruction sent by the host computer, the second image under the acquisition scene written in the last time period is read from the memory at a reading speed supported by the memory, and the writing step is returned to be executed, that is, the reading and writing of the memory are multiplexed by alternately reading and writing, so that the image transmission meets the requirement of high frame rate, and the image transmission speed is improved.

[0159] In one embodiment, the read-write control module 806 is configured to store the first image transmitted through the image transmission protocol into the first-in-first-out module at a writing speed supported by the memory, read the first image from the first-in-first-out module, and write the first image into the memory; and the host computer data interface module 802 is configured to transmit the second image to the host computer through the first-in-first-out module.

[0160] In the embodiment, the first image transmitted through the image transmission protocol is stored into the first-in-first-out module at a writing speed supported by the memory, the first image is read from the first-in-first-out module and written into the memory, the target image is transmitted to the host computer through the first-in-first-out module, the clock synchronization can be maintained through the first-in-first-out module, and the image data overflow can be avoided when the channel is blocked during the transmission of the second image to the host computer.

[0161] In one embodiment, the image transmission device further comprises a frame filling module configured to generate a frame filling frame based on preset data agreed with the host computer when the second image is lost frames; the host computer data interface module 802 is configured to transmit the frame filling frame to the host computer; and the frame filling frame is used to instruct the host computer to send a first reset instruction for resetting the image acquisition device and the master chip.

[0162] and the first reset instruction is used to reset the image acquisition device and the master chip based on the first reset instruction.

[0163] In the embodiment, when the frame is lost, the frame filling frame is generated based on preset data agreed with the host computer, and the frame filling frame is transmitted to the host computer, so that the collapse of the host computer data transmission interface is avoided, the image transmission time is long, and the image acquisition device may have a bug or the master chip may have a bug in the case of the frame loss failure. By receiving the first reset instruction sent by the host computer, the image acquisition device and the master chip are reset, the fault is solved, and the image transmission efficiency is improved.

[0164] In one embodiment, the image transmission device further comprises a row filling module configured to generate a row filling frame based on the read second image and preset data agreed with the host computer when the second image is lost rows;

[0165] The host computer data interface module 802 is configured to transmit the row filling frame to the host computer; and the row filling frame is used to instruct the host computer to send a second reset instruction for resetting the master chip.

[0166] and the second reset instruction is used to reset the master chip based on the second reset instruction.

[0167] In the embodiment, when a line is lost, a line loss completion frame is generated based on preset data agreed with the host computer, and the line loss completion frame is transmitted to the host computer, so that the collapse of the data transmission interface of the host computer is avoided, and the image transmission time is long; in the case of line loss failure, the main control chip is likely to have a bug, and the main control chip is reset by receiving the second reset instruction sent by the host computer, so that the failure can be solved, and the image transmission efficiency is improved.

[0168] In one embodiment, the image transmission device further comprises a bit width conversion module, configured to perform bit width conversion on the first image transmitted through the image transmission protocol to obtain an image with a bit width adapted to the memory;

[0169] The read-write control module 806 is configured to write the first image with a bit width adapted to the memory into the memory;

[0170] The bit width conversion module is configured to perform bit width conversion on the second image read from the memory to obtain a second image with a bit width adapted to the data interface of the host computer;

[0171] The host computer data interface module 802 is configured to transmit the second image adapted to the data interface of the host computer through the data interface of the host computer.

[0172] In the embodiment, the first image transmitted through the image transmission protocol is subjected to bit width conversion to obtain an image with a bit width adapted to the memory, the first image with a bit width adapted to the memory is written into the memory, the second image read from the memory is subjected to bit width conversion to obtain a second image with a bit width adapted to the data of the host computer, and the second image adapted to the data interface of the host computer is transmitted through the data interface of the host computer, so that a large number of images originally stored in the host computer can be stored in the memory of the peripheral device, and the image transmission efficiency is improved.

[0173] In one embodiment, the acquisition scene includes at least one of a color control scene of light, a brightness control scene of light, a laser type control scene, and a light supplement lamp control scene.

[0174] In the embodiment, the color control of light, the brightness control of light, the laser type control, and the light supplement lamp control can be used to obtain images in different acquisition scenes for testing, calibration, and other processing, to meet various requirements, and the image transmission efficiency is improved to reduce the delay caused by switching between acquisition scenes.

[0175] The specific limitations of the image transmission device can refer to the limitations of the image transmission method described above, which will not be repeated here. Each module in the above image transmission device can be implemented by software, hardware and their combination. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor calls and executes the operations corresponding to the above modules.

[0176] A master control chip is characterized in that the master control chip is used to implement the steps of the above method embodiments.

[0177] In one embodiment, a computer readable storage medium is provided, and the computer program is stored on the computer readable storage medium. The computer program is executed by the processor to implement the steps of the above method embodiments.

[0178] In one embodiment, a computer program product or computer program is provided, and the computer program product or computer program includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the steps in the above method embodiments.

[0179] A person of ordinary skill in the art can understand that all or part of the above-mentioned method embodiments can be completed by a computer program to instruct related hardware. The computer program can be stored in a non-volatile computer readable storage medium. The computer program can include the above-mentioned method embodiments when executed. Any reference to memory, storage, database or other medium in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0180] The above only describes the preferred embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. An image transmission method characterized by, The method is applied to a master control chip, and the method comprises the following steps: receiving a collection scene and an image collection instruction sent by a host computer; obtaining a first image collected based on the image collection instruction and in the collection scene; writing the first image transmitted through an image transmission protocol into a memory at a writing speed supported by the memory; the writing speed is greater than a transmission speed of the image transmission protocol; receiving an image continuous reading instruction sent by the host computer, and reading a second image of a target collection scene written in a previous time period from the memory at a reading speed supported by the memory; returning to the step of writing the first image transmitted through the image transmission protocol into the memory at the writing speed supported by the memory until a reading stop instruction sent by the host computer is detected, and stopping reading the second image of the target collection scene from the memory; transmitting the second image to the host computer.

2. The method of claim 1, wherein, The step of writing the first image transmitted through the image transmission protocol into the memory at the writing speed supported by the memory comprises the following steps: storing the first image transmitted through the image transmission protocol into a first-in-first-out module, and reading the first image from the first-in-first-out module and then writing the first image into the memory at the writing speed supported by the memory; The step of transmitting the second image to the host computer comprises the following step: transmitting the second image to the host computer through the first-in-first-out module.

3. The method of claim 1, wherein, The method further comprises the following steps: when the second image is lost frames, generating a lost frame completion frame based on preset data agreed with the host computer; transmitting the lost frame completion frame to the host computer; the lost frame completion frame is used to instruct the host computer to send a first reset instruction for resetting an image collection device and the master control chip; receiving the first reset instruction, and resetting the image collection device and the master control chip based on the first reset instruction.

4. The method of claim 1, wherein, The method further comprises the following steps: when the second image is lost lines, generating a lost line completion frame based on the second image that has been read and preset data agreed with the host computer; transmitting the lost line completion frame to the host computer; the lost line completion frame is used to instruct the host computer to send a second reset instruction for resetting the master control chip; receiving the second reset instruction, and resetting the master control chip based on the second reset instruction.

5. The method of claim 1, wherein, The step of writing the first image transmitted through the image transmission protocol into the memory comprises the following steps: performing bit width conversion on the first image transmitted through the image transmission protocol to obtain a first image with a bit width adapted to the memory; writing the first image with the bit width adapted to the memory into the memory; The step of transmitting the second image to the host computer comprises the following steps: performing bit width conversion on the second image read from the memory to obtain a second image with a bit width adapted to a data interface of the host computer; transmitting the second image adapted to the data interface of the host computer to the host computer through the data interface of the host computer.

6. The method according to any one of claims 1 to 5, characterized in that, The collection scene comprises at least one of a color control scene of light, a brightness control scene of light, a laser type control scene and a light supplement lamp control scene.

7. An image transmission apparatus characterized by comprising: The device comprises: The host computer data interface module is configured to receive a collection scene and an image collection instruction sent by the host computer. The image analysis module is configured to acquire a first image collected based on the image collection instruction and in the collection scene. The read-write control module is configured to write the first image into a memory. The host computer data interface module is configured to receive an image reading instruction sent by the host computer. The read-write control module is configured to read a second image in a target collection scene from the memory based on the image reading instruction. The host computer data interface module is configured to transmit the second image to the host computer. The image transmission device is configured to implement the steps of the method according to any one of claims 1 to 6.

8. A master chip, characterized by The master control chip is configured to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images