Endoscope data processing method, neural network architecture, device and equipment

Abstract

The invention relates to the technical field of data processing, and discloses an endoscope data processing method, a neural network architecture, a device and equipment. The method comprises the following steps: acquiring image data acquired by an endoscope; extracting an image feature frame of the image data, and performing data processing on the image feature frame by adopting multi-level storage in a preset neural network to obtain target image processing data; the architecture of the preset neural network is a sparse storage architecture; and performing data adjustment on the target image processing data to obtain target image data. By implementing the invention, the data processing speed and the computing power utilization rate of the processing chip are improved, and the data transmission capability of a chip communication line is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a method for processing endoscopic data, a neural network architecture, a device, and a device. Background Art

[0002] An endoscopic assisted diagnosis system is a diagnosis system that uses artificial intelligence to assist in detecting diseases with an endoscope. It can achieve high accuracy in the case of a large number of endoscopic disease types and diverse lesion manifestations, and can achieve stable diagnosis in a highly complex clinical operation environment of human-machine cooperation.

[0003] In the related art, the computing power of an endoscopic assisted diagnosis system mainly depends on the multiplier resources of the selected processing chip. Usually, the processing unit (DSP) accounts for a relatively limited proportion in the chip, and in the artificial intelligence (AI) algorithm, due to operations such as activation functions, fixed-point quantization, and weight structure pruning, the sparsity of image features and weight coefficients is likely to increase, resulting in serious waste of chip computing power. Or, a processing chip with stronger computing power is selected for data processing, but such a chip usually has a fixed computing architecture and cannot achieve an update speed that matches the algorithm. Summary of the Invention

[0004] In view of this, the present invention provides a method for processing endoscopic data, a neural network architecture, a device, and a device to solve the problems of insufficient computing power and serious waste of the processing chip.

[0005] In a first aspect, the present invention provides a method for processing endoscopic data, including: acquiring image data collected by an endoscope; extracting an image feature frame of the image data, and performing data processing on the image feature frame by using multi-level storage in a preset neural network to obtain target image processing data; the architecture of the preset neural network is a sparse storage architecture; and performing data adjustment on the target image processing data to obtain target image data.

[0006] The method for processing endoscopic data provided by the embodiments of the present invention, after acquiring the image data collected by the endoscope, processes the extracted image feature frame based on the multi-level storage and sparse storage architecture in the preset neural network to obtain target image data. Among them, through the multi-level storage in the neural network, the ability of the communication line of the processing chip to transmit data is improved, and the transmission delay is reduced. At the same time, the sparse storage architecture effectively saves the storage space of the processing chip and improves the utilization rate of computing power.

[0007] In an alternative embodiment, the preset neural network includes a first storage unit, the first storage unit corresponding to three-level storage, and the target image processing data includes first image processing data; extracting an image feature frame of the image data, and performing data processing on the image feature frame by using the multi-level storage in the preset neural network to obtain the target image processing data, including: obtaining intermediate data of the image feature frame; storing the intermediate data based on the first-level storage corresponding to the first storage unit; arranging the intermediate data based on the second-level storage corresponding to the first storage unit to generate an arrangement result corresponding to the intermediate data; performing parallel processing on the intermediate data according to the arrangement result to generate a parallel processing result; and performing a zero-removing operation on the parallel processing result by using data sparsity based on the third-level storage corresponding to the first storage unit to obtain the first image processing data.

[0008] In the method for processing endoscopic data provided by the embodiment of the present invention, after obtaining the intermediate data of the image feature frame, the multi-level storage in the preset neural network performs data processing on the intermediate data to obtain the first image processing data, thereby improving the effective computing power of the processing chip and the ability of the communication line to transmit data.

[0009] In an alternative embodiment, the preset neural network further includes a second storage unit, the second storage unit corresponding to three-level storage, and the target image processing data includes second image processing data; extracting an image feature frame of the image data, and performing data processing on the image feature frame by using the multi-level storage in the preset neural network to obtain the target image processing data, further including: obtaining the architecture weight parameters of the preset neural network; compressing and storing the architecture weight parameters based on the first-level storage corresponding to the second storage unit; extracting the target weight parameters required currently from the compressed and stored architecture weight parameters based on the second-level storage corresponding to the second storage unit; and caching the second image processing data based on the third-level storage corresponding to the second storage unit.

[0010] In the method for processing endoscopic data provided by the embodiment of the present invention, after obtaining the architecture weight parameters of the preset neural network, the multi-level storage in the preset neural network performs data processing on the architecture weight parameters to obtain the second image processing data, thereby effectively improving the computing power utilization rate of the chip and providing the characteristics of low latency and high throughput on the premise of ensuring the stability and reliability of the system.

[0011] In an alternative embodiment, performing parallel processing on the intermediate data according to the arrangement result to generate a parallel processing result includes: performing parallel processing on the intermediate data according to the arrangement result and the target weight parameters to obtain a parallel processing result, and the parallel processing result is the second image processing data.

[0012] The endoscopic data processing method provided by the embodiments of the present invention performs parallel processing on intermediate data based on the arrangement result and the target weight parameter, thereby improving the data operation processing rate.

[0013] In an alternative embodiment, it further includes: fusing the first image processing data and the second image processing data to obtain target image processing data.

[0014] The endoscopic data processing method provided by the embodiments of the present invention performs multi-module processing on the image processing data and then fuses it, ensuring the accuracy of the image processing data and facilitating the acceleration processing of the acquired image processing data.

[0015] In a second aspect, the present invention provides a neural network architecture for executing the endoscopic data processing method of the first aspect or any corresponding embodiment thereof. The neural network architecture includes: at least one first processing chip for acquiring the image data collected by the endoscope and extracting image feature frames from the image data; a second processing chip communicatively connected to at least one first processing chip, the second processing chip including a multi-level storage unit for performing sparse storage processing on the data frames through the multi-level storage unit to generate target image data.

[0016] The neural network architecture provided by the embodiments of the present invention uses the first processing chip to acquire the image data collected by the endoscope, and uses the second processing chip to perform sparse storage processing on the extracted image feature frames based on the multi-level storage unit in the preset neural network to obtain target image data. Thereby, the data transmission capacity of the processing chip communication line is improved, the data transmission delay is reduced, the storage space of the processing chip is saved, and the computing power utilization rate of the chip is improved.

[0017] In an alternative embodiment, the multi-level storage unit includes a first storage unit and a second storage unit; the first storage unit is used to access the image feature frames through multi-level storage; the second storage unit is used to access the target weight parameter through multi-level storage.

[0018] The neural network architecture provided by the embodiments of the present invention uses the first storage unit and the second storage unit to store and access the image feature frames and the target weight parameter respectively, thereby improving the effective computing power of the processing chip, improving the data transmission capacity of the communication line, and effectively improving the computing power utilization rate of the chip on the premise of ensuring the stable and reliable operation of the system.

[0019] In an alternative embodiment, the neural network architecture further includes: a serial transceiver unit disposed on the first processing chip, and at least one first processing chip is interconnected through the serial transceiver unit.

[0020] In the neural network architecture provided by the embodiments of the present invention, multiple first processing chips can be interconnected through a serial transceiver unit, that is, a distributed computing architecture that supports multi-chip interconnection, thereby greatly improving the data processing speed of the processing chips.

[0021] In a third aspect, the present invention provides a processing device for endoscopic data, including: an acquisition module for acquiring image data collected by an endoscope; a processing module for extracting an image feature frame of the image data and performing data processing on the image feature frame by using multi-level storage in a preset neural network to obtain target image processing data; the architecture of the preset neural network is a sparse storage architecture; and an adjustment module for performing data adjustment on the target image processing data to obtain target image data.

[0022] In a fourth aspect, the present invention provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the method for processing endoscopic data according to the first aspect or any corresponding embodiment thereof.

[0023] In a fifth aspect, the present invention provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to cause a computer to execute the method for processing endoscopic data according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0025] Figure 1 is a schematic flowchart of a method for processing endoscopic data according to an embodiment of the present invention;

[0026] Figure 2 is a schematic flowchart of another method for processing endoscopic data according to an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of a neural network architecture according to an embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of another neural network architecture according to an embodiment of the present invention;

[0029] Figure 5 is a schematic diagram of yet another neural network architecture according to an embodiment of the present invention;

[0030] Figure 6 is a block diagram of a processing device for endoscopic data according to an embodiment of the present invention;

[0031] Figure 7 is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed implementation manners

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0033] The endoscopic assisted diagnosis system is a diagnosis system that uses artificial intelligence for assisted detection by an endoscope, mainly used for monitoring and processing the captured images during endoscopic operation. Further, it can perform real-time monitoring on the endoscopic video images, standardize the operations of medical staff, and reduce the occurrence of missed diagnoses and misdiagnoses. At the same time, in the case of a large variety of diseases and diverse lesion manifestations, high accuracy can be achieved, and stable diagnosis can be realized in a highly complex clinical practical environment of human-machine collaboration.

[0034] In the related art, the computing power of the endoscopic assisted diagnosis system mainly depends on the selected processing chip. Usually, the proportion of the DSP unit in the chip is relatively limited, and in the AI algorithm, due to operations such as activation functions, fixed-point quantization, and weight structure pruning, the sparsity of image features and weight coefficients is likely to increase, that is, the number of zeros in the operation process increases. Multiplying by zero is an invalid calculation and wastes the chip computing power, resulting in serious waste of the chip computing power and an increase in the delay of AI inference of the endoscopic assisted diagnosis system. Or, a processing chip with stronger computing power is selected for data processing, such as an ASIC chip. Although the ASIC chip can achieve strong computing power, it usually has a fixed computing architecture and it is difficult to achieve an update speed that matches the algorithm.

[0035] Based on this, the technical solution of the present invention improves the data transmission capacity of the communication line of the processing chip and reduces the data transmission delay by setting a multi-level storage architecture in the neural network. At the same time, data processing is performed based on the neural network with a sparse storage architecture, effectively saving the storage space of the processing chip and improving the computing power utilization rate of the chip.

[0036] According to an embodiment of the present invention, an embodiment of a method for processing endoscopic data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0037] In this embodiment, a method for processing endoscopic data is provided, which can be used in a computer device, such as an endoscopic processing device, in which a neural network architecture generated based on a sparse storage structure is deployed. Figure 1 It is a flowchart of a method for processing endoscopic data according to an embodiment of the present invention, as Figure 1 shown, and this process includes the following steps:

[0038] Step S101, obtain the image data collected by the endoscope.

[0039] The image data is an image or video of a human target part collected by the endoscope, such as oral cavity image data, bronchial image data, abdominal cavity image data, etc. The endoscope is communicatively connected to the computer device, and the endoscope can transmit the image data it collects to the computer device. Correspondingly, the computer device can obtain the image data captured by the endoscope for subsequent processing.

[0040] Step S102, extract the image feature frame of the image data, and perform data processing on the image feature frame by using multi-level storage in a preset neural network to obtain target image processing data.

[0041] The image feature frame is a representative image frame in the image data, that is, an image frame with a high information content. Specifically, computer vision and image processing techniques can be used here to extract the image feature frame in the image data.

[0042] The target image processing data is the image data output by a preset neural network. The preset neural network is a neural network pre-trained based on a sparse storage architecture, and multi-level storage, such as three-level storage, is set in the preset neural network. In the process of performing data processing on the image feature frame, using a multi-level storage architecture can optimize the data access process to effectively improve the data stream bandwidth.

[0043] Due to calculations such as the commonly used activation function (RELU) in the convolutional neural network (CNN), a large number of zero values will be generated during the calculation process, as well as during model optimization and fixed-point quantization. The zero values will waste a large amount of storage memory and computing resources of the processing chip. Here, a multi-level sparse storage architecture is adopted, and the sparsity of different data is used to perform data processing on the image feature frame, improving the computing power utilization rate of each processing chip in the neural network.

[0044] Step S103: Adjust the target image processing data to obtain target image data.

[0045] The target image data is the endoscopic image data finally required for display. During the process of generating the target image processing data, the image data is adjusted to adapt to the preset neural network. Therefore, after obtaining the target image processing data, it is necessary to adjust the obtained target image processing data to output target image data that meets the display requirements. Specifically, the data adjustment may include summarization of image processing data, format adjustment of image processing data, vector calculation of image processing data, etc. The specific method of data adjustment is not specifically limited here, and those skilled in the art can determine it according to actual needs.

[0046] The method for processing endoscopic data provided by the embodiments of the present invention, after acquiring the image data collected by the endoscope, processes the extracted image feature frames based on the multi-level storage and sparse storage architecture in the preset neural network to obtain target image data. Among them, through the multi-level storage in the neural network, the ability of the communication line of the processing chip to transmit data is improved, and the transmission delay is reduced. At the same time, the sparse storage architecture effectively saves the storage space of the processing chip and improves the utilization rate of computing power.

[0047] In this embodiment, a method for processing endoscopic data is provided, which can be used in a computer device, such as an endoscopic processing device, in which a neural network architecture generated based on a sparse storage structure is deployed. Figure 2 It is a flowchart of another method for processing endoscopic data according to an embodiment of the present invention, as Figure 2 shown, and this process includes the following steps:

[0048] Step S201: Acquire the image data collected by the endoscope. For details, please refer to Figure 1 Step S101 of the embodiment shown, which will not be elaborated here.

[0049] Step S202: Extract the image feature frames of the image data, and use the multi-level storage in the preset neural network to process the image feature frames to obtain target image processing data.

[0050] Specifically, the preset neural network includes a first storage unit, the first storage unit corresponds to three-level storage, the target image processing data includes first image processing data, and the above step S202 includes:

[0051] Step S2021: Acquire the intermediate data of the image feature frames.

[0052] Intermediate data refers to the intermediate-layer data generated during the data processing process. For example, temporary calculation results, cleaned image feature frame data, sorting results of image feature frames, etc. The endoscope processing device can monitor the data processing process of the image feature frames in real time to obtain the intermediate data generated during the data processing process, and control the conversion, processing, and transmission of the intermediate data in the neural network.

[0053] Step S2022: Store the intermediate data based on the primary storage corresponding to the first storage unit.

[0054] Use the primary storage corresponding to the first storage unit to store the intermediate data generated during the data processing process. Among them, the primary storage can be a primary buffer (buffer).

[0055] Step S2023: Arrange the intermediate data based on the secondary storage corresponding to the first storage unit to generate an arrangement result corresponding to the intermediate data.

[0056] Reconstruct the structure of the intermediate data using the secondary storage corresponding to the first storage unit, that is, arrange the intermediate data according to the type of data calculation and processing to generate an arrangement result corresponding to the intermediate data. Among them, the types of data calculation and processing include tensor type and vector type. Among them, the secondary storage can be a secondary buffer.

[0057] Step S2024: Based on the tertiary storage corresponding to the first storage unit, use data sparsity to perform a zero-removing operation on the parallel processing result to obtain the first image processing data.

[0058] The tertiary storage corresponding to the first storage unit is constructed by reg_array, and the zero-removing operation of multiplication is performed on the parallel processing result using the sparsity of the data.

[0059] Furthermore, the preset neural network further includes a second storage unit. The second storage unit corresponds to the tertiary storage, and the target image processing data includes the second image processing data. The above step S202 further includes:

[0060] Step S2025: Obtain the architecture weight parameters of the preset neural network.

[0061] The architecture weight parameters are the weights pre-configured for each network parameter in the preset neural network. The architecture weight parameters are the optimal parameter values for data processing obtained during the training of the preset neural network. Specifically, the architecture weight parameters can be quantized here to obtain parameter values of the corresponding data type. For example, after quantization, int8-type architecture weight parameters are obtained.

[0062] Step S2026: Compress and store the architecture weight parameters based on the primary storage corresponding to the second storage unit.

[0063] Input the architecture weight parameters of the preset neural network into the primary storage corresponding to the second storage unit for compression. Here, the primary storage can be a double data rate synchronous dynamic random access memory (DDR). Thus, all the architecture weight parameters in the preset neural network are stored in the primary storage, facilitating subsequent calls and accesses.

[0064] Step S2027: Based on the secondary storage corresponding to the second storage unit, extract the target weight parameters required currently from the compressed and stored architecture weight parameters.

[0065] When the preset neural network involves convolution operations in the convolution (conv2D) layer or data connection in the fully connected (FC) layer during the data processing process, in order to achieve the accuracy of convolution operations and the data connection accuracy between different layers, the corresponding architecture weight parameters need to be retrieved.

[0066] Specifically, as described above, all the architecture weight parameters are compressed and stored in the primary storage, and a corresponding index address is assigned to each parameter for subsequent retrieval. The target weight parameters are the architecture weight parameters required during the current data processing process. The required target weight parameters are selected from the compressed and stored weight parameters through the index and read into the secondary storage of the second storage unit.

[0067] It should be noted that during the data processing process, since multiplying zero by any value results in zero, which is an invalid calculation, it is necessary to select non-zero architecture weight parameters through the address index and read them into the secondary storage.

[0068] Step S2028: Based on the tertiary storage corresponding to the second storage unit, cache the second image processing data.

[0069] Use the tertiary storage corresponding to the second storage unit to cache the second image processing data to ensure the uninterruptedness of data processing.

[0070] In some alternative embodiments, the above method further includes: performing parallel processing on the intermediate data according to the arrangement result corresponding to the intermediate data and the target weight parameters to obtain a parallel processing result, and the parallel processing result is the second image processing data.

[0071] According to the arrangement result corresponding to the intermediate data and the target weight parameters, input the intermediate data into different processing elements (PEs) respectively to perform parallel processing on the intermediate data, saving the data processing time. For example Figure 4As shown, the intermediate data output by the first storage unit (feature_buf) and the target weight parameters output by the second storage unit (kernel_disp) are input into each computing unit PE to implement parallel processing of the intermediate data and obtain the second image processing data.

[0072] In the above embodiment, the intermediate data is processed in parallel based on the arrangement result and the target weight parameters, thereby improving the rate of data operation processing.

[0073] In an alternative embodiment, the above step S202 may further include:

[0074] Step S2029, fusing the first image processing data and the second image processing data to obtain the target image processing data.

[0075] After obtaining the first image processing data and the second image processing data, the first image processing data and the second image data are fused to generate the target image processing data. Specifically, a weight coefficient for image processing data fusion can be preset here to perform weighted fusion on the first image processing data and the second image data to obtain the corresponding target image processing data.

[0076] The endoscopic data processing method provided by the embodiments of the present invention performs multi-module processing on the image processing data and then fuses it, ensuring the accuracy of the image processing data and facilitating the acceleration processing of the acquired image processing data.

[0077] Step S203, perform data adjustment on the target image processing data to obtain the target image data. For details, please refer to Figure 1 Step S103 of the embodiment shown, which will not be elaborated here.

[0078] The endoscopic data processing method provided by the embodiments of the present invention, after obtaining the intermediate data of the image feature frame, presets multi-level storage in the neural network to process the intermediate data and obtain the first image processing data, thereby improving the effective computing power of the processing chip and the ability of the communication line to transmit data. After obtaining the architecture weight parameters of the preset neural network, the multi-level storage in the preset neural network is used to process the architecture weight parameters to obtain the second image processing data, thereby effectively improving the computing power utilization rate of the chip and providing the characteristics of low latency and high throughput on the premise of ensuring system stability and reliability.

[0079] In this embodiment, a neural network architecture is also provided, as Figure 3 shown, including:

[0080] At least one first processing chip, configured to obtain the image data collected by the endoscope and extract the image feature frame from the image data.

[0081] The image data collected at the front end is transmitted into the high-speed transceiver (GTH) through an interface, such as a SERDES interface. Since the endoscope is relatively large in size, it is necessary to first pass through an adjustment module (resize) to adjust the collected image data to the fixed output size set by the network model, and then extract the image feature frame from the image data (frame_process).

[0082] In addition, the neural network parameters are configured into the internal memory (BRAM) of the processing chip using the USB interface, and the architecture weight parameters are configured into the DDR at the same time. The base address and storage size are configured by the central processing unit (CPU) system.

[0083] A second processing chip is communicatively connected to at least one first processing chip. The second processing chip includes a multi-level storage unit, and sparse storage processing is performed on the data frame through the multi-level storage unit to generate target image data.

[0084] The second processing chip includes multiple tensor processing units (TPUs) and supports parallel or serial deployment of multiple TPUs for processing the extracted image feature frames. That is, the operator part of the image feature frame is implemented on the TPU, and the operator part may include a convolution operator, a pooling operator, a fully connected operator, but is not limited thereto.

[0085] Furthermore, the second processing chip includes a multi-level storage unit, and sparse storage processing is performed on the data frame through the multi-level storage unit.

[0086] The neural network architecture provided by the embodiments of the present invention uses the first processing chip to obtain the image data collected by the endoscope, and uses the second processing chip to perform sparse storage processing on the extracted image feature frames based on the multi-level storage unit in the preset neural network to obtain the target image data. Thereby, the ability of the communication line of the processing chip to transmit data is improved, the latency of transmitting data is reduced, the storage space of the processing chip is saved, and the computing power utilization rate of the chip is improved.

[0087] In a common convolutional neural network (CNN) acceleration system, when the absolute computing power provided by the multiply-accumulate unit (MAC) is relatively fixed, the bandwidth provided by the system data path can determine the effective computing power of the system. Therefore, the neural network architecture in this embodiment uses multi-level storage to optimize the image feature frames and target weight parameters.

[0088] In some alternative embodiments, as Figure 4 shown, the above multi-level storage unit includes: a first storage unit, a second storage unit. runtime_schedue is the system scheduling and monitoring module, which is responsible for the overall operation of the system.

[0089] The first storage unit is used to access the image feature frames through multi-level storage.

[0090] After the image feature frames cache the intermediate data in the first-level storage (feature_buf), they are passed to the second-level storage. The second-level storage arranges the data according to the data type and stores the data into line_buf_0 and line_buf_1 respectively for ping-pong operation.

[0091] The second storage unit is used to access the target weight parameters through multi-level storage.

[0092] As described above, the architecture weight parameters are pre-configured and can be stored in ddr_ctrl. All architecture weight parameters enter the multi-level storage in the second storage unit kernel_disp through ddr_ctrl. Among them, the first-level storage compresses and stores the weight parameters; the second-level storage selects and caches the current required target weight parameters from the first-level storage according to the address index.

[0093] The intermediate data and the target weight parameters are simultaneously passed to the computing units (PE_0 - PE_31) according to the output channels to complete data calculation. After the calculated data is aggregated, format adjusted (mergereg), and vector calculated (active&softmax), the data is output by out_ctrl.

[0094] The neural network architecture provided by the embodiments of the present invention uses the first storage unit and the second storage unit to store and access the image feature frames and the target weight parameters respectively, thereby improving the effective computing power of the processing chip, enhancing the ability of the communication line to transmit data, and effectively improving the computing power utilization rate of the chip on the premise of ensuring the stable and reliable operation of the system.

[0095] In some alternative embodiments, as Figure 5 shown, the above neural network architecture further includes: a serial transceiver unit, which is arranged on the first processing chip, and at least one first processing chip is interconnected through the serial transceiver unit.

[0096] The neural network architecture may include multiple processing chips, such as FPGA【0, 0】, FPGA【0, 1】, FPGA【1, 0】, FPGA【1, 1】. Each processing chip is configured with four high-speed transceivers (GTH), two of which are used to receive the data sent by the adjacent two processing chips, and two are used to send data to the adjacent two processing chips.

[0097] For example, among the four GTHs of the FPGA [0, 1], the GTH located above the chip sends data to the FPGA [0, 0], the GTH located on the right side of the chip sends data to the FPGA [1, 1], the GTH located on the left side of the chip is used to receive the data sent by the FPGA [1, 1], and the GTH located below the chip is used to receive the data sent by the FPGA [0, 0]. That is, the neural network architecture in this embodiment supports a distributed computing architecture with multi-chip interconnection, and the neural network can be split and deployed to different processing chips in the above manner to complete the interaction between data.

[0098] The neural network architecture provided by the embodiment of the present invention enables multiple first processing chips to be interconnected through a serial transceiver unit, that is, it supports a distributed computing architecture with multi-chip interconnection, thereby greatly improving the data processing speed of the processing chips.

[0099] In this embodiment, a processing device for endoscopic data is also provided. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0100] This embodiment provides a processing device for endoscopic data, as Figure 6 shown, including:

[0101] An acquisition module 301, configured to acquire image data collected by an endoscope;

[0102] A processing module 302, configured to extract an image feature frame of the image data, perform data processing on the image feature frame by using multi-level storage in a preset neural network to obtain target image processing data; the architecture of the preset neural network is a sparse storage architecture;

[0103] An adjustment module 303, configured to perform data adjustment on the target image processing data to obtain target image data.

[0104] In some optional implementation manners, the processing module 302 includes:

[0105] A first acquisition sub-module, configured to acquire intermediate data of the image feature frame;

[0106] A first storage sub-module, configured to store the intermediate data based on the first-level storage corresponding to the first storage unit;

[0107] A first generation sub-module, configured to arrange the intermediate data based on the second-level storage corresponding to the first storage unit to generate an arrangement result corresponding to the intermediate data;

[0108] A second generation sub-module, configured to perform parallel processing on intermediate data according to the arrangement result to generate a parallel processing result;

[0109] A processing sub-module, configured to perform a zero-removing operation on the parallel processing result based on the three-level storage corresponding to the first storage unit by utilizing data sparsity to obtain first image processing data.

[0110] In some alternative embodiments, the processing module 302 further includes:

[0111] A second acquisition sub-module, configured to acquire architecture weight parameters of a preset neural network.

[0112] A second storage sub-module, configured to perform compressed storage on the architecture weight parameters based on the primary storage corresponding to the second storage unit;

[0113] An extraction sub-module, configured to extract target weight parameters required currently from the compressed-stored architecture weight parameters based on the secondary storage corresponding to the second storage unit;

[0114] A cache sub-module, configured to cache second image processing data based on the three-level storage corresponding to the second storage unit.

[0115] In some alternative embodiments, the second generation sub-module includes:

[0116] A processing unit, configured to perform parallel processing on intermediate data according to the arrangement result and the target weight parameters to obtain a parallel processing result, where the parallel processing result is second image processing data.

[0117] In some alternative embodiments, the processing module 302 further includes:

[0118] A fusion sub-module, configured to fuse the first image processing data and the second image processing data to obtain target image processing data.

[0119] The further function descriptions of the above-mentioned various modules and units are the same as those in the corresponding above embodiments, and will not be elaborated here.

[0120] The processing device for endoscopic data in this embodiment is presented in the form of functional units. Here, the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0121] The processing device for endoscopic data provided by the embodiments of the present invention and the processing method for endoscopic data provided by the embodiments of the present invention, after acquiring the image data collected by the endoscope, process the extracted image feature frames based on the multi-level storage and sparse storage architectures in the preset neural network to obtain the target image data. Among them, through the multi-level storage in the neural network, the data transmission capacity of the communication line of the processing chip is improved, and the transmission delay is reduced. At the same time, the sparse storage architecture effectively saves the storage space of the processing chip and improves the utilization rate of computing power.

[0122] The embodiments of the present invention further provide a computer device having the above-mentioned Figure 6 processing device for endoscopic data shown.

[0123] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of a computer device provided by an optional embodiment of the present invention. As Figure 7 shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common main board or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (such as an array of servers, a set of blade servers, or a multi-processor system). Figure 7 Here, one processor 10 is taken as an example.

[0124] The processor 10 can be a central processing unit, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above-mentioned hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above-mentioned programmable logic device can be a complex programmable logic device, a field programmable gate array, a general array logic, or any combination thereof.

[0125] Among them, the memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiments.

[0126] The memory 20 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the computer device and the like. In addition, the memory 20 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 may optionally include a memory remotely provided with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0127] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, a hard disk, or a solid-state drive; the memory 20 may further include a combination of the above types of memory.

[0128] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30, and the output device 40 can be connected through a bus or other means. Figure 7 Taking connection through a bus as an example.

[0129] The input device 30 can receive input digital or character information, and generate key signal inputs related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (such as an LED), and a tactile feedback device (such as a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light-emitting diode, a display, and a plasma display. In some alternative embodiments, the display device may be a touch screen.

[0130] Embodiments of the present invention also provide a computer-readable storage medium. The methods according to the embodiments of the present invention can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented as computer code that is originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the methods described herein can be stored as such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by the computer, the processor, or the hardware, the methods shown in the above embodiments are implemented.

[0131] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for processing endoscope data, characterized in that: The method comprises: Acquiring image data collected by the endoscope; Extracting an image feature frame of the image data, and performing data processing on the image feature frame using a multi-level storage in a preset neural network to obtain target image processing data; the architecture of the preset neural network is a sparse storage architecture; The target image processing data is subjected to data adjustment to obtain target image data.

2. The method according to claim 1, characterized in that The preset neural network includes a first storage unit, the first storage unit corresponds to a third-level storage, and the target image processing data includes first image processing data; The step of extracting the image feature frame of the image data and performing data processing on the image feature frame using multi-level storage in a preset neural network to obtain target image processing data includes: Acquire intermediate data of the image feature frame; Storing the intermediate data based on the primary storage corresponding to the first storage unit; Arrange the intermediate data based on the secondary storage corresponding to the first storage unit to generate an arrangement result corresponding to the intermediate data; Process the intermediate data in parallel according to the arrangement result to generate a parallel processing result; Based on the tertiary storage corresponding to the first storage unit, a zero removal operation is performed on the parallel processing result by utilizing data sparsity to obtain first image processing data.

3. The method according to claim 2, characterized in that The preset neural network further includes a second storage unit, the second storage unit corresponds to the third-level storage, and the target image processing data includes second image processing data; The extracting of the image feature frame of the image data, and performing data processing on the image feature frame using the multi-level storage in the preset neural network to obtain the target image processing data also includes: Obtaining architecture weight parameters of the preset neural network; Compressing and storing the architecture weight parameter based on the primary storage corresponding to the second storage unit; Based on the secondary storage corresponding to the second storage unit, extracting the currently required target weight parameter from the compressed and stored architecture weight parameter; Based on the tertiary storage corresponding to the second storage unit, the second image processing data is cached.

4. The method according to claim 2, characterized in that: The performing parallel processing on the intermediate data according to the arrangement result to generate a parallel processing result includes: According to the arrangement result and the target weight parameter, the intermediate data is processed in parallel to obtain a parallel processing result, and the parallel processing result is the second image processing data.

5. The method according to claim 3, characterized in that: Also includes: The first image processing data and the second image processing data are fused to obtain the target image processing data.

6. A neural network architecture, characterized in that: For executing the method for processing endoscopic data according to any one of claims 1 to 5, the neural network architecture comprises: at least one first processing chip, used to obtain image data collected by the endoscope, and extract the image feature frame from the image data; The second processing chip is communicatively connected to the at least one first processing chip, and the second processing chip includes a multi-level storage unit, through which the image feature frame is sparsely stored to generate target image data.

7. The neural network architecture according to claim 6, characterized in that The multi-level storage unit includes a first storage unit and a second storage unit; The first storage unit is used to access the image feature frame through multi-level storage; The second storage unit is used to access the target weight parameter through multi-level storage.

8. The neural network architecture according to claim 6, characterized in that Also includes: A serial transceiver unit is provided on the first processing chip, and the at least one first processing chip is interconnected through the serial transceiver unit.

9. An endoscope data processing device, characterized in that: The device comprises: An acquisition module, used for acquiring image data collected by the endoscope; A processing module, used for extracting an image feature frame of the image data, and performing data processing on the image feature frame using a multi-level storage in a preset neural network to obtain target image processing data; the architecture of the preset neural network is a sparse storage architecture; The adjustment module is used to perform data adjustment on the target image processing data to obtain target image data.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for processing endoscopic data according to any one of claims 1 to 5 by executing the computer instructions.

Images