Image feature data classification method and electronic equipment

Abstract

The invention provides an image feature data classification method and electronic equipment, and belongs to the technical field of data classification processing. The method is applied to a field programmable gate array (FPGA), and each data processing module in the FPGA runs in a parallel assembly line mode. The method comprises the following steps: performing feature extraction on original image data to obtain initial target feature data and background non-feature data; preprocessing the feature data to obtain target feature data and a corresponding label; classifying the target feature data according to the label to obtain multiple groups of target feature data and the current cycle index thereof, and calculating the initial address of each group of target feature data according to the label, the current cycle index and the corresponding pre-stored space capacity; sequentially writing each group of target feature data into the target storage space according to the initial address; and sequentially transmitting the various target data features to a rear end according to a preset transmission mode. The image feature data classification processing method and device can achieve the effect of improving the image feature data classification processing speed and efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of data classification and processing, and in particular to an image feature data classification method and an electronic device. Background Art

[0002] With the rapid development of intelligent electronic devices, people's requirements for image resolution and transmission frame rates are becoming increasingly higher. However, the development of high-resolution and high-frame-rate images has also led to an explosive growth in image data. In some technical fields, such as security monitoring, medical imaging, and autonomous driving, there are high requirements for real-time image data processing.

[0003] Currently, the process from acquiring raw image data from front-end sensors to ultimately transmitting the processed image results to the back-end requires multiple steps in image data processing. Among these steps, target feature data extraction and classification, a key step in image data processing, requires automated recognition and classification of raw image data based on visual features. This is then balanced with the accuracy of image feature extraction and classification through pre-built and deployed hardware accelerators.

[0004] However, when extracting and classifying target features from raw image data using related technologies, frequent access to the image data results in significant delays in image processing results. Furthermore, a large amount of temporary iterative data needs to be cached, which overloads the hardware accelerator and reduces image processing efficiency. Consequently, these related technology solutions suffer from low image data processing efficiency and poor real-time performance. Summary of the Invention

[0005] The purpose of this application is to provide an image feature data classification method and electronic device, which can improve the speed and efficiency of image feature data classification processing.

[0006] The embodiment of the present application is implemented as follows: According to a first aspect of an embodiment of the present application, a method for classifying image feature data is provided. The method is applied to a field programmable gate array (FPGA). The FPGA includes an extraction module, a preprocessing module, a real-time classification module, a classification storage module, and an output module. The extraction module, the preprocessing module, the real-time classification module, the classification storage module, and the output module are operated in a parallel pipeline manner. The method includes: The extraction module obtains the current frame raw image data collected by the front-end sensor, and performs feature extraction on the current frame raw image data to obtain initial target feature data and background non-feature data, and sends the initial target feature data and background non-feature data to the preprocessing module; The preprocessing module preprocesses the initial target feature data and background non-feature data to obtain target feature data and labels corresponding to the target feature data; The real-time classification module classifies the target feature data according to the labels corresponding to the target feature data to obtain multiple groups of target feature data and the current cycle number of each group of target feature data, and calculates the starting address of each group of target feature data according to the labels corresponding to each group of target feature data, the current cycle number of each group of target feature data, and the pre-stored space capacity corresponding to each group of target feature data, and generates a data packet corresponding to each group of target feature data; The classification storage module writes the data packets corresponding to each group of target characteristic data into the target storage space in the external memory in sequence according to the starting address of each group of target characteristic data; The output module reads the data packets corresponding to each set of target feature data from the target storage space, and parses each data packet to obtain various target feature data corresponding to the original image of the current frame, and transmits various target data features in sequence to the back-end server according to the preset transmission method.

[0007] As a possible implementation method, the initial target feature data and background non-feature data are preprocessed to obtain target feature data and labels corresponding to the target feature data, including: Marking the current initial target feature data according to the current initial target feature data and the associated feature data of the current initial target feature data to obtain a label corresponding to the current initial target feature data; The background non-feature data is filtered according to the label corresponding to the initial target feature data to obtain the target feature data and the label corresponding to the target feature data.

[0008] As a possible implementation method, the target feature data is classified according to the labels corresponding to the target feature data to obtain multiple groups of target feature data and the current cycle number of each group of target feature data, including: Determining whether the feature type of the target feature data has changed based on the tag corresponding to the current target feature data and the tag corresponding to the pre-read next target feature data; If yes, stop reading the next target feature data, and take the current target feature data and the target feature data read before the current target feature data as a set of target feature data, and count the current cycle times of the current target feature data; If not, continue to read the next target feature data until the feature type of the target feature data changes.

[0009] As a possible implementation method, based on the label corresponding to each set of target feature data, the current number of cycles of each set of target feature data, and the pre-stored space capacity corresponding to each set of target feature data, the starting address of each set of target feature data is calculated, and a data packet corresponding to each set of target feature data is generated, including: Calculating the product of the label corresponding to each set of target feature data and the capacity of the pre-stored space corresponding to each set of target feature data; Calculate the sum of the product and the current number of cycles of each set of target feature data to obtain the starting address of each set of target feature data; An end code is filled after the last target feature data of each group of target feature data to obtain a data packet corresponding to each group of target feature data.

[0010] As a possible implementation method, according to the starting address of each group of target feature data, the data packets corresponding to each group of target feature data are sequentially written into the target storage space in the external memory, including: According to the starting address of each group of target feature data, the position of the target storage space corresponding to each group of target feature data in the external memory is determined, and each group of target feature data is written into the target storage space in sequence.

[0011] As a possible implementation method, classifying the target feature data according to the labels corresponding to the target feature data also includes: Enable the label corresponding to each target feature data in the current frame original image data.

[0012] As a possible implementation method, data packets corresponding to each set of target feature data are read from the target storage space, and each data packet is parsed to obtain various target feature data corresponding to the current frame original image, and each target data feature is sequentially transmitted to the back-end server according to a preset transmission method, including: Read the data packets corresponding to each set of target feature data from the target storage space according to the enable flag, parse each data packet in real time, and find the position of the end code to determine whether the target feature data of the current feature type has been read; If so, the current target feature data is transmitted to the backend server according to the transmission method corresponding to the target feature data of the current feature type.

[0013] As a possible implementation method, feature extraction is performed on the original image data of the current frame to obtain initial target feature data and background non-feature data, including: Edge detection processing is performed on the original image data of the current frame to obtain the initial target feature data and background non-feature data corresponding to the original image data of the current frame, and feature extraction is performed on the initial target feature data to determine the basic information of the original image data of the current frame, which includes: coordinate information and center line information.

[0014] As a possible implementation, the above-mentioned image feature data classification method further includes: Before receiving the next frame of original image data, the auxiliary information of the current frame of original image data is deleted, the auxiliary information including: the label, the number of cycles, the characteristic information and the starting address of each target feature data in the current frame of original image.

[0015] According to a second aspect of an embodiment of the present application, an electronic device is provided, which includes a field programmable gate array (FPGA), and the FPGA is used to execute the steps of the image feature data classification method described in the first aspect.

[0016] The beneficial effects of the embodiments of the present application include: An embodiment of the present application provides an image feature data classification method, which realizes image data classification through an extraction module, a preprocessing module, a real-time classification module, a classification storage module and an output module running in a parallel pipeline in a field programmable gate array FPGA. Among them, the extraction module performs feature extraction on the current frame original image data transmitted by the front-end sensor at the current moment to obtain the initial target feature data and background non-feature data corresponding to the specific target specified by the user, and synchronously transmits the initial target feature data and background non-feature data to the preprocessing module; the preprocessing module calibrates the initial target feature data and filters out the isolated initial target features in the background non-feature data to obtain target feature data and labels corresponding to the target feature data, and sends the target feature data and its corresponding labels to the real-time classification module; the real-time classification module classifies the target feature data according to the labels of the target feature data to obtain multiple groups of target feature data and the labels of each group of target feature data in this round of image feature classification. The starting address of each target feature data group is calculated based on the number of cyclic occurrences before data classification, the number of cyclic occurrences of each target feature data group's label before the current round of image feature data classification, and the capacity of the pre-stored storage space corresponding to each target feature data group, and a data packet corresponding to each target feature data group is generated. The classification storage module sequentially writes the data packets corresponding to each target feature data group into the target storage space in the external memory based on the starting address of each target feature data group. After the target feature data of the current frame's original image data is classified, the output module sequentially reads all target feature data corresponding to each specific target from the target storage space and transmits them to the backend server in real time according to a preset output method. The parallel pipeline operation of the modules in the field programmable gate array (FPGA) can accelerate the image data classification process and alleviate FPGA memory pressure through the external memory. Furthermore, the target feature data label not only distinguishes the specific target category to which it belongs, but also reduces noise in background non-feature data, filtering out isolated initial target feature data mixed in the background non-feature data, thereby improving the precision and accuracy of target feature data extraction. In this way, the speed and efficiency of image feature data classification can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A system architecture diagram of a field programmable gate array (FPGA) provided in an embodiment of the present application; Figure 2 A flowchart of the first image feature data classification method provided in an embodiment of the present application; Figure 3 A flowchart of a second method for classifying image feature data provided in an embodiment of the present application; Figure 4 A flowchart of a third method for classifying image feature data provided in an embodiment of the present application; Figure 5 A schematic diagram of writing target characteristic data into a target storage space provided in an embodiment of the present application; Figure 6 A flowchart of a fifth image feature data classification method provided in an embodiment of the present application; Figure 7 An example diagram of an image feature data classification method provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0021] Currently, in traditional image data processing, the lack of parallelism in image data processing structures means that both target feature extraction and classification require frequent access to image data. This results in increased time consumption, significant latency in the output of image data processing results, and poor real-time performance. Furthermore, the target feature extraction and classification process requires a large amount of temporary iterative data to be cached to complete the target feature data classification. This places an excessive load on the hardware accelerator, reducing the efficiency of image data processing.

[0022] To this end, an embodiment of the present application provides an image feature data classification method. By constructing a field programmable gate array (FPGA), image feature data classification processing is implemented. The FPGA is deployed with an extraction module, a preprocessing module, a real-time classification module, a classification storage module, and an output module that operate in a parallel pipeline manner. An external memory is connected to the FPGA to store temporary iterative target feature data. The FPGA performs noise reduction on the original image data of the current frame through the extraction module and labels the target feature data through the preprocessing module. The real-time classification module classifies the data according to the label of the target feature data and dynamically calculates the starting address of each target feature data. The classification storage module writes each target feature data into the corresponding target storage space in the external memory according to the starting address of the target feature data. The output module reads the target feature data in each target storage space in turn. In this way, the speed and efficiency of image feature data classification processing can be improved.

[0023] The image feature classification acceleration method and electronic device provided in the embodiments of the present application are explained in detail below with reference to the accompanying drawings.

[0024] Figure 1 For a system architecture diagram of a field programmable gate array FPGA provided in this application, see Figure 1 The field programmable gate array (FPGA) provided in the embodiment of the present application includes: an extraction module, a preprocessing module, a real-time classification module, a classification storage module, and an output module. Among them, the classification storage module is connected to an external memory to reduce the data cache pressure of the FPGA.

[0025] Furthermore, the input end of the extraction module is connected to the front-end server, and the extraction module is used to obtain original image data from the front-end server, and extract background feature data and target feature data from the original image data, and transmit the extracted target feature data and background feature data to the preprocessing module; the preprocessing module performs secondary filtering on the received target feature data and background feature data, and marks the filtered target feature data, and transmits the target feature data and its corresponding mark to the real-time classification module; the real-time classification module classifies and enables marking the target feature data based on the label, and calculates the starting address of the target feature data based on the preliminary classification result, and transmits the classification result and the starting address of each target feature data to the classification storage module; the classification storage module writes the target feature data into the target storage space in the external memory according to the starting address of the target feature data, and the output module reads each data packet in the target storage space in turn to realize the transmission of the classified feature output to the back end.

[0026] Exemplarily, after the field programmable gate array FPGA receives the original image A, the extraction module performs feature extraction on image A in a left-to-right and top-to-bottom order. After the extraction module extracts the first row of image feature data a1 of the original image A, it obtains the initial target feature data and background invalid data corresponding to the first row of image feature data a1, and sends the initial target feature data and background invalid data corresponding to the first row of image feature data a1 to the preprocessing module. The preprocessing module starts to perform secondary filtering and labeling on the initial target feature data and background invalid data corresponding to the first row of image feature data a1 to obtain the target feature data and label corresponding to the first row of image feature data a1, and transmits them to the real-time classification module for classification; at the same time, the extraction module synchronously starts to scan and extract the initial target feature data and background invalid data corresponding to the second row of image feature data a2 of the original image A, and sends the initial target feature data and background invalid data corresponding to the second row of image feature data a2 to the preprocessing module, and so on, until the last column of image feature data of the original image A is processed.

[0027] It is worth noting that the extraction module processes the received original image data in the order from left to right and then from top to bottom, or from top to bottom and then from left to right. This application does not make any specific restrictions on this.

[0028] Optionally, the field programmable gate array FPGA can be specifically implemented by a programmable chip, a programmable semiconductor device, etc., and this application does not make any specific limitation on this.

[0029] Optionally, the internal processing logic of the extraction module, preprocessing module, classification module, classification storage module, and output module within the field-programmable gate array (FPGA) can be adjusted based on user needs and application scenarios. Specifically, when the extraction module extracts the first piece of initial image feature data and transmits it to the preprocessing module for secondary filtering and labeling, the extraction module simultaneously processes the second piece of initial image data, and so on, until the original image data received from the front-end sensor is processed. In other words, when any module within the FPGA performs image data processing operations, other modules outside of that module also run synchronously to perform their functions, and each module operates in parallel.

[0030] Among them, the front-end sensor can be an image acquisition device such as a camera, a pinhole camera, an infrared camera, etc., and the back-end PC can be an electronic device, a computer device, a tablet computer, etc. Terminal devices, this application does not make specific restrictions on this.

[0031] In this way, in the entire image classification processing flow of the field programmable gate array FPGA, each module operates in parallel pipeline, which can realize real-time processing and real-time transmission of image data, greatly improving the speed and efficiency of image data classification, and solving the problem of poor real-time performance of traditional image data classification.

[0032] Figure 2 This is a flowchart of an image feature data classification method provided by this application, see Figure 2 , this method is applied to Figure 1 In the field programmable gate array FPGA shown, the image feature data classification method provided by the embodiment of the present application includes: S201, the extraction module obtains the current frame original image data collected by the front-end sensor, and performs feature extraction on the current frame original image data to obtain initial target feature data and background non-feature data, and sends the initial target feature data and background non-feature data to the preprocessing module.

[0033] Optionally, the extraction module can obtain the original image data collected by the front-end sensor at the current moment from the front-end sensor through a serial interface, I / O interface or other communication interface. The original image can be a CT image, a road environment image, etc. This application does not make specific limitations on this.

[0034] Optionally, the current frame raw image data refers to a frame of raw image data sent by the front-end sensor to the field programmable gate array FPGA for image data extraction and classification at the current moment. The frequency at which the front-end sensor sends raw image data to the field programmable gate array FPGA can be set according to the application scenario of the front-end sensor and the back-end customer needs. This application does not make any specific limitations on this.

[0035] Optionally, after acquiring the raw image data of the current frame, the extraction module invokes a global search algorithm pre-programmed into the extraction module to perform a full-area scan of the raw image data of the current frame to identify a specific target, and then performs feature extraction on the raw image data of the current frame to obtain initial target feature data and background non-feature data corresponding to the raw image data of the current frame. It is worth noting that when the extraction module performs a full-area scan of the raw image data of the current frame, the feature extraction sequence may be from left to right, then from top to bottom, or other scanning and extraction sequences, and this application does not impose specific limitations on this.

[0036] Among them, the specific target refers to the target object that the user wants, such as a specific person or object in a photo. The specific target that the user wants to query in the current frame original image data can be one or more, and this application does not make specific restrictions on this.

[0037] Optionally, when the extraction module performs a global scan on the original image data of the current frame, it also extracts feature data of the original image data of the current frame. The global scanning algorithm can be implemented by an edge detection program, which is not specifically limited in this application.

[0038] Exemplarily, after the extraction module receives the original image A sent by the front-end sensor at the current moment, the extraction module performs binary image processing on the original image A as a whole, divides the original image A into initial target feature data and background non-feature data, calibrates the initial target feature data to 1, and calibrates the background non-feature data to 0 to obtain the binary image data after edge detection.

[0039] Optionally, the initial target feature data refers to the image feature data that is preliminarily identified in the original image data of the current frame as possibly belonging to a specific target, and the background non-feature data refers to the image feature data other than the image feature data of the specific target in the original image data of the current frame, that is, the background non-feature data is regarded as image feature data that is unrelated to the specific target.

[0040] Furthermore, after the extraction module extracts the initial target feature data and background non-feature data from the original image data of the current frame, it synchronously transmits them to the preprocessing module for the next step of image processing.

[0041] S202: The preprocessing module preprocesses the initial target feature data and the background non-feature data to obtain target feature data and labels corresponding to the target feature data.

[0042] Optionally, the preprocessing module preprocesses the received initial target feature data and background non-feature data, further removing interfering image feature data to obtain target feature data belonging to a specific target, and labels each target feature data according to the characteristic target to which the target feature data belongs, thereby obtaining a label for each target feature data. The label is mainly used to distinguish the specific target object to which the target feature data belongs and does not provide any important information.

[0043] For example, if the specific targets that the user wants in the current frame original image A are circle 1, circle 2 and circle 3, the extraction module scans the first row of image data a1 of the original image data A from left to right to obtain an image feature sequence such as oooxxxpppxxoxqqq. The extraction module calibrates the extracted image feature sequence to obtain a binary data sequence of 1110001110010111. The extraction module synchronously transmits the calibrated binary data sequence and its corresponding image feature sequence to the preprocessing module. The preprocessing module processes the image feature sequence according to the specific targets specified by the user and the binary data sequence. The initial target feature data marked as 1 in the image feature sequence is labeled according to the sequence. For example, the target feature data corresponding to circle 1 is labeled with label number m, the target feature data corresponding to circle 2 is labeled with label number n, and the target feature data corresponding to circle 3 is labeled with label number k. Then, the three initial target feature data of ooo are all labeled with m, the three initial target feature data of ppp are all labeled with n, the initial target feature data of o in xxox is labeled with m, and the three initial target feature data of qqq are all labeled with k, to obtain the label sequence of mmm000nnn00m0kkk. At the same time, the preprocessing module further determines whether the isolated feature in 00m0 is the target feature data based on the corresponding label of the initial target image data and the relevance of the specific target, and deletes this isolated interference signal, ultimately obtaining the target feature data sequence of ooo-ppp-qqq and its corresponding label sequence of mmm-nnn-kkk. The target feature data sequence and its corresponding label sequence are then sent to the real-time classification module.

[0044] It is worth noting that the preprocessing module marks the corresponding labels on the initial target feature data based on the data continuity or correlation of the specific target selected by the user. The data continuity is used to indicate the correlation between the pixels or spatial positions of the initial target feature data and its previous and subsequent initial target feature data.

[0045] Optionally, after the extraction module processes the current frame of raw image data, it does not stop running, but synchronously receives the next frame of raw image data of the current frame of raw image and performs feature extraction on the next frame of raw image data, and so on, until the last frame of raw image data is processed.

[0046] Among them, the label values ​​of the target feature data corresponding to different specific targets are different, and the label values ​​can be set arbitrarily, such as the label of the target feature data of circle 1 is 1, the label corresponding to the target feature data of circle 2 is 10, etc. This application does not make specific limitations on this.

[0047] S203. The real-time classification module classifies the target feature data according to the labels corresponding to the target feature data to obtain multiple groups of target feature data and the current number of cycles of each group of target feature data. The real-time classification module calculates the starting address of each group of target feature data according to the labels corresponding to each group of target feature data, the current number of cycles of each group of target feature data, and the pre-stored space capacity corresponding to each group of target feature data, and generates a data packet corresponding to each group of target feature data.

[0048] Optionally, the current number of cycles refers to the number of times the label of each group of target feature data has appeared in a cycle before the current target feature data classification. For example, if the target feature data o is classified for the fourth time, the number of cycles in which the label m corresponding to the target feature data o in the first three data classifications has appeared is counted.

[0049] Optionally, the real-time classification module classifies the target feature data according to the labels corresponding to the target feature data to obtain multiple groups of target feature data, and counts the number of times the labels of each target feature data appear cyclically before the current moment. Each target feature data in each group of target feature data belongs to the image features of the same specific target, and each target feature data in each group of target feature data has the same label.

[0050] Exemplarily, after the real-time classification module receives the target feature data sequence ooo-ppp-qqq and its corresponding label sequence mmm-nnn-kkk, the real-time classification module divides the target feature data sequence ooo-ppp-qqq into three groups of target feature data according to the label sequence mmm-nnn-kkk, and obtains the historical number of cycles of label m, label n, and label k before the current moment. The historical number of cycles of label m before this round of target feature data classification is 10, the historical number of cycles of label n before this round of target feature data classification is 15, and the historical number of cycles of label k before this round of target feature data classification is 5.

[0051] Optionally, the pre-stored space capacity is the storage space size that the user pre-allocates in the external memory for the image feature data of a specific target based on the attributes of the specific target. The pre-stored space capacity can be 1000, 10000, etc., which is specifically determined by the data characteristics of the specific target. This application does not make any specific restrictions on this.

[0052] Optionally, the starting address of each set of target feature data is calculated based on the label corresponding to each set of target feature data, the number of times the label of each set of target feature data appears in a cycle before the current round of image feature data classification, and the capacity of the pre-stored space corresponding to each set of target feature data. The starting address is specifically used to indicate the starting point where the target feature data is written to the external memory. It is worth noting that the starting address of each set of target feature data is different. For the same specific target, the starting address of different sets of target feature data is also different. The real-time classification module can flexibly calculate the corresponding storage starting position based on the order in which each set of target feature data is processed.

[0053] Furthermore, to ensure the integrity of each group of target feature data written into the external memory by the classification storage module, the real-time classification module fills an end code after the last target feature data in each group of target feature data to form a complete data packet structure.

[0054] S204 , the classification storage module writes the data packets corresponding to each group of target feature data into the target storage space in the external memory in sequence according to the starting address of each group of target feature data.

[0055] Optionally, the classification storage module writes the data packets of each group of target feature data into the target storage space at the corresponding position in the external memory according to the starting address of each group of target feature data, wherein the target storage space is a storage space pre-divided by the user for the image feature data of a specific target.

[0056] S205. The output module reads the data packets corresponding to each group of target feature data from the target storage space, and parses each data packet to obtain various target feature data corresponding to the current frame original image, and transmits various target data features in sequence to the back-end server according to the preset transmission method.

[0057] Optionally, after all target feature data in the current frame of raw image data has been processed, the field programmable gate array (FPGA) initiates a data read operation, and the output module sequentially reads the complete target feature data corresponding to each specific target from each target storage space in the external memory. The output module determines whether the target feature data for each specific target has been read based on whether an end code has been read.

[0058] Furthermore, the output module retrieves the corresponding transmission protocol according to the image feature attributes of each specific target, and transmits all the complete target feature data of the specific target to the back-end server.

[0059] It is worth noting that image data classification is specifically performed for different specific targets. The image feature data that the field programmable gate array FPGA ultimately transmits to the PC is the image feature data corresponding to each specific target. For example, the target image feature data sequence of circle 1 and the target feature data sequence of circle 2 are transmitted to the PC in sequence.

[0060] In an embodiment of the present application, image data classification is achieved through the extraction module, preprocessing module, real-time classification module, classification storage module and output module running in parallel pipeline in the field programmable gate array FPGA. Among them, the extraction module performs feature extraction on the current frame original image data transmitted by the front-end sensor at the current moment to obtain the initial target feature data and background non-feature data corresponding to the specific target specified by the user, and synchronously transmits the initial target feature data and background non-feature data to the preprocessing module; the preprocessing module calibrates the initial target feature data and filters out the isolated initial target features in the background non-feature data to obtain target feature data and labels corresponding to the target feature data, and sends the target feature data and its corresponding labels to the real-time classification module; the real-time classification module classifies the target feature data according to the labels of the target feature data to obtain multiple groups of target feature data and the labels of each group of target feature data in this round of image feature. The starting address of each target feature data group is calculated based on the number of cyclic occurrences before data classification, the number of cyclic occurrences of each target feature data group's label before the current round of image feature data classification, and the capacity of the pre-stored storage space corresponding to each target feature data group, and a data packet corresponding to each target feature data group is generated. The classification storage module sequentially writes the data packets corresponding to each target feature data group into the target storage space in the external memory based on the starting address of each target feature data group. After the target feature data of the current frame's original image data is classified, the output module sequentially reads all target feature data corresponding to each specific target from the target storage space and transmits them to the backend server in real time according to a preset output method. The parallel pipeline operation of the modules in the field programmable gate array (FPGA) can accelerate the image data classification process and alleviate FPGA memory pressure through the external memory. Furthermore, the target feature data label not only distinguishes the specific target category to which it belongs, but also reduces noise in background non-feature data, filtering out isolated initial target feature data mixed in the background non-feature data, thereby improving the precision and accuracy of target feature data extraction. In this way, the speed and efficiency of image feature data classification can be improved.

[0061] In an optional embodiment, see Figure 3 The operation of step S202 may specifically be: S301: Label the current initial target feature data according to the current initial target feature data and the associated feature data of the current initial target feature data to obtain a label corresponding to the current initial target feature data.

[0062] Optionally, since the image feature data of the same specific target have a certain continuity or correlation in space, the initial target feature data is marked with corresponding labels based on the correlation between the image feature data of the same specific target, so that the target feature data belonging to the same specific target are all marked with the same label.

[0063] Specifically, the current initial target feature data is used as the base point data, and the feature data around the current initial target feature data is used as the reference point to determine whether the current initial target feature data and the adjacent initial target feature data belong to the same specific target. Based on the judgment result, a custom label is added to the initial target feature data to distinguish the image feature data of different specific targets.

[0064] S302: Filter the background non-feature data according to the label corresponding to the initial target feature data to obtain the target feature data and the label corresponding to the target feature data.

[0065] Optionally, the background non-feature data is filtered according to the label corresponding to the initial target feature data, and some isolated initial target feature data mixed in the background non-feature data is deleted, thereby improving the accuracy and precision of the target feature data.

[0066] Optionally, the target feature data refers to the initial target feature data remaining after deleting the isolated initial target feature data, and the label of the target feature data remains unchanged.

[0067] Furthermore, the target feature data is sequentially placed into an internal buffer in the order in which it is processed, awaiting the next step of processing by the real-time classification module. The pre-processing module reads and stores the target feature data simultaneously, ensuring the parallel operation of the image feature data classification process. It also serves as an isolation and buffer to prevent target feature data loss when the back-end server is busy.

[0068] In an optional embodiment, see Figure 4 The operation of "classifying the target feature data according to the labels corresponding to the target feature data to obtain multiple groups of target feature data and the current cycle times of each group of target feature data" in step S203 may specifically be: S401 : Determine whether the feature type of the target feature data has changed based on a tag corresponding to the current target feature data and a tag corresponding to the pre-read next target feature data.

[0069] Optionally, the current target feature data refers to the target feature data that the real-time classification module currently reads from the preprocessing module and is ready to write into the real-time classification module's own cache area, and the pre-read next target feature data refers to the target feature data that is ready to be written into the real-time classification module's own cache area at the next moment.

[0070] Optionally, based on a comparison result between a tag corresponding to the pre-read next target feature data and a tag of the current target feature data, it is determined whether the next target feature data and the current target feature data belong to the same specific target.

[0071] S402: If yes, stop reading the next target feature data, and take the current target feature data and the target feature data read before the current target feature data as a group of target feature data, and count the current cycle times of the current target feature data.

[0072] Optionally, if the label of the pre-read next target feature data is different from the label of the current target feature data, that is, the next target feature data and the current target feature data do not belong to the same specific target, then the reading of the next target feature data is paused, and the current target feature data is used as the last target feature data of the current group of target feature data, and the target feature data with the same label previously read is used as the same group of target feature data, and the number of times the label corresponding to the current target feature data has appeared in a cycle before this round of image feature classification processing is counted.

[0073] S403: If not, continue to read the next target feature data until the feature type of the target feature data changes.

[0074] Optionally, if the label of the pre-read next target feature data is the same as the label of the current target feature data, that is, the next target feature data and the current target feature data belong to the same specific target, then continue to read the next target feature data until the label of the next target feature data changes, and then execute step S402.

[0075] In an optional embodiment, see Figure 4 The operation of "calculating the starting address of each set of target feature data and generating a data packet corresponding to each set of target feature data according to the label corresponding to each set of target feature data, the current number of cycles of each set of target feature data, and the pre-stored space capacity corresponding to each set of target feature data" in step S203 may specifically be: S404, calculating the product of the label corresponding to each set of target feature data and the pre-stored space capacity corresponding to each set of target feature data; calculating the sum of the product and the current cycle number of each set of target feature data to obtain the starting address of each set of target feature data.

[0076] Optionally, the real-time classification module calculates the starting address of each group of target feature data in real time according to the following formula (1), which is as follows: D=U×C+cnt(1) Among them, D is used to indicate the starting address of each group of target feature data, U is used to indicate the value of the label corresponding to each group of target feature data, C is used to indicate the preset space capacity corresponding to each group of target feature data, and cnt is used to indicate the number of times the label of each group of target feature data appears in the cycle before this round of image feature data classification processing.

[0077] For example, if the label number m corresponding to the target feature data o is 10, the preset space capacity C corresponding to the target feature data o is 10000, and the number of cycles cnt of the target feature data o before this round of image feature data classification processing is 100, then the starting address of this group of target feature data o during this round of image feature data classification processing is D=10×10000+100=100100.

[0078] S405 , filling an end code after the last target feature data of each group of target feature data to obtain a data packet corresponding to each group of target feature data.

[0079] Optionally, an end code is added after the last target feature data in each set of target feature data to form a data packet corresponding to each set of target feature data. It is worth noting that the end code added after each set of target feature data belonging to a specific target is the same. After being written into the target storage space, the added end codes are automatically merged.

[0080] Figure 5 This is a schematic diagram of writing target feature data into target storage space provided by this application, see Figure 5 , the internal cache area A refers to the cache area in the preprocessing module, and the internal cache area B refers to the cache area in the real-time classification module. The preprocessing module reads the initial target feature data and background non-feature data from the internal cache area A of the extraction module, and after preprocessing the initial target feature data and background non-feature data, writes the target feature data into the internal cache area B in the order of reading, and writes each group of classified target feature data into the target storage space in the external memory DCR in real time according to the calculated starting address.

[0081] In an optional implementation manner, the operation of step S204 may specifically be: According to the starting address of each group of target feature data, the position of the target storage space corresponding to each group of target feature data in the external memory is determined, and each group of target feature data is written into the target storage space in sequence.

[0082] Optionally, the external memory is divided into multiple storage spaces according to the attributes of each specific target, each specific target corresponds to a storage space, and the classification storage module writes each group of target feature data into the corresponding target storage space according to the starting address of each group of target feature data.

[0083] If the target feature data corresponding to a specific target exceeds the capacity of the target storage space allocated therefor, the classified storage space may delete part of the target feature data to ensure that the target feature data written into the target storage space does not overflow.

[0084] It is worth noting that each set of target feature data corresponding to the same specific target should be written into the same target storage space.

[0085] In an optional implementation manner, the real-time classification module in the field programmable gate array may also be used to: Enable the label corresponding to each target feature data in the current frame original image data.

[0086] Optionally, each target feature data in the original image data of the current frame is enabled and marked, such as all target feature data with label m in the original image data of the current frame are enabled and pulled high, so that the output module can only read the target feature data with the enable mark, thereby improving the reading speed of the classification feature data.

[0087] In an optional embodiment, see Figure 6 The operation of step S205 may specifically be: S601: Read data packets corresponding to each group of target feature data from the target storage space according to the enable flag, parse each data packet in real time, and find the position of the end code to determine whether the target feature data of the current feature type has been read.

[0088] Optionally, the output module reads the data packets of target characteristic data corresponding to each specific target from the target storage space in the external memory in sequence according to the enable mark, and parses the data packets in real time until the end code is read, thereby determining that the target characteristic data corresponding to the current specific target has been read.

[0089] It is worth noting that when the output module parses the data packet, the data of the data packet will be temporarily stored in the buffer area of ​​the output module, which can prevent data loss caused by busy transmission lines.

[0090] S602: If yes, transmit the current target feature data to the backend server according to the transmission mode corresponding to the target feature data of the current feature type.

[0091] Optionally, the current feature type is used to indicate the type of the specific target corresponding to the current target feature data, and the corresponding communication protocol is called according to the attributes of the current specific target to transmit the target feature data to the PC.

[0092] In an optional implementation manner, the operation of step S201 may specifically be: Edge detection processing is performed on the original image data of the current frame to obtain initial target feature data and background non-feature data corresponding to the original image data of the current frame, and feature extraction is performed on the initial target feature data to determine basic information of the original image data of the current frame, the basic information including: coordinate information and center line information.

[0093] Optionally, an edge detection algorithm is used to process the raw image data of the current frame to obtain multiple initial target feature data 1 and multiple background non-feature data 0 in the raw image data of the current frame. Point-to-point feature extraction is then performed on the initial target feature data 1 to obtain coordinate information and centerline information of the raw image data of the current frame. The basic information of the raw image data of the current frame can be used for three-dimensional image fitting on the backend server. The coordinate information refers to the coordinates of each target feature point in the raw image of the current frame, and the centerline information refers to the plane where the centerline of the raw image of the current frame lies.

[0094] In an optional implementation, the above-mentioned image feature data classification method further includes: Before receiving the next frame of original image data, the auxiliary information of the current frame of original image data is deleted, the auxiliary information including: the label, the number of cycles, the characteristic information and the starting address of each target feature data in the current frame of original image.

[0095] Optionally, before receiving the next frame of raw image data, all auxiliary information of the current frame of raw image data cached in the field programmable gate array FPGA is deleted, which can ensure that the extraction and classification processing of the next frame of raw image data is not affected by the current frame of raw image data.

[0096] Figure 7 This is an example diagram of an image feature data classification method provided by this application, see Figure 7The specific implementation steps of the image feature data classification method provided in the embodiment of the present application are as follows: S1, the extraction module in the field programmable gate array FPGA receives the current frame original image data Y from the front-end sensor, and performs feature extraction on the current frame original image data Y to obtain the image feature 000aa00aabbb0bbb00000cccc00ccc00a0aaa0aabb0bb000 corresponding to Y, and sends the image feature to the preprocessing module; S2, the preprocessing module processes the image feature transmitted by the extraction module to obtain a labeled target feature data sequence aaaabbbbbbcccccccaaaaaabbbbb, and sends the target feature data sequence and its label to the real-time classification module; S3, the real-time classification module divides the target feature data sequence aaaabbbbbbcccccccaaaaaabbbbb into five groups of target feature data a according to the label aaa-bbbbbb-ccccccc-aaaaaa-bbbbb, and generate data packets aaaa+ data packet, bbbbbb+ data packet, cccccc+ data packet, aaaaaa+ data packet and bbbbb+ data packet corresponding to each group of target feature data and transmit them to the classification storage module; S4, the classification storage module writes each group of target feature data into the external memory DCR in turn, so that each target storage space stores the complete data packets aaaaaaaaaaa+ data packet, bbbbbbbbbbb+ data packet and ccccccc+ data packet of each type of target feature data; S5, after the output module completes the classification of all image feature data of the current frame original image data Y, it reads various types of target feature data aaaaaaaaaaa, bbbbbbbbbbb, ccccccc from the external memory via the classification storage module, and the classification processing of the current frame original image data Y is completed.

[0097] Figure 8 This is a schematic diagram of the structure of an electronic device provided in this application, see Figure 8 The electronic device provided in the embodiment of the present application includes Figure 1 The field programmable gate array FPGA shown, the electronic device implements the steps of the above-mentioned image feature data classification method via the field programmable gate array FPGA. Its specific implementation process and technical effects can be found in the above-mentioned specific embodiments, and this application will not go into details here.

[0098] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0099] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for classifying image feature data, characterized in that: The method is applied to a field programmable gate array (FPGA), wherein the FPGA includes: an extraction module, a preprocessing module, a real-time classification module, a classification storage module, and an output module. The extraction module, the preprocessing module, the real-time classification module, the classification storage module, and the output module are operated in a parallel pipeline manner. The method includes: The extraction module obtains the current frame raw image data collected by the front-end sensor, performs feature extraction on the current frame raw image data to obtain initial target feature data and background non-feature data, and sends the initial target feature data and background non-feature data to the preprocessing module; The preprocessing module preprocesses the initial target feature data and the background non-feature data to obtain target feature data and a label corresponding to the target feature data; The real-time classification module classifies the target feature data according to the labels corresponding to the target feature data to obtain multiple groups of target feature data and the current cycle number of each group of target feature data, and calculates the starting address of each group of target feature data according to the labels corresponding to each group of target feature data, the current cycle number of each group of target feature data, and the pre-stored space capacity corresponding to each group of target feature data, and generates a data packet corresponding to each group of target feature data; The classification storage module writes the data packets corresponding to each group of target feature data into the target storage space in the external memory in sequence according to the starting address of each group of target feature data; The output module reads the data packets corresponding to each group of target feature data from the target storage space, and parses each data packet to obtain various target feature data corresponding to the current frame original image, and transmits the various target data features in sequence to the back-end server according to a preset transmission method.

2. The image feature data classification method according to claim 1, characterized in that: The preprocessing of the initial target feature data and the background non-feature data to obtain target feature data and labels corresponding to the target feature data includes: Marking the current initial target feature data according to the current initial target feature data and the associated feature data of the current initial target feature data to obtain a label corresponding to the current initial target feature data; The background non-feature data is filtered according to the label corresponding to the initial target feature data to obtain target feature data and the label corresponding to the target feature data.

3. The image feature data classification method according to claim 1, characterized in that: The target feature data is classified according to the labels corresponding to the target feature data to obtain multiple groups of target feature data and the current cycle number of each group of target feature data, including: Determining whether the feature type of the target feature data has changed based on the tag corresponding to the current target feature data and the tag corresponding to the pre-read next target feature data; If yes, stop reading the next target feature data, and take the current target feature data and the target feature data read before the current target feature data as a set of target feature data, and count the current cycle times of the current target feature data; If not, continue to read the next target feature data until the feature type of the target feature data changes.

4. The image feature data classification method according to claim 1, characterized in that: The method of calculating the starting address of each set of target feature data according to the label corresponding to each set of target feature data, the current number of cycles of each set of target feature data, and the pre-stored space capacity corresponding to each set of target feature data, and generating a data packet corresponding to each set of target feature data, includes: Calculating the product of the label corresponding to each set of target feature data and the capacity of the pre-stored space corresponding to each set of target feature data; Calculating the sum of the product and the current number of cycles of each set of target feature data to obtain the starting address of each set of target feature data; An end code is filled after the last target feature data of each group of target feature data to obtain a data packet corresponding to each group of target feature data.

5. The image feature data classification method according to claim 1, characterized in that: The step of sequentially writing data packets corresponding to each set of target characteristic data into a target storage space in the external memory according to the starting address of each set of target characteristic data comprises: According to the starting address of each group of target feature data, the position of the target storage space corresponding to each group of target feature data in the external memory is determined, and each group of target feature data is written into the target storage space in sequence.

6. The image feature data classification method according to claim 1, characterized in that: The classifying the target feature data according to the label corresponding to the target feature data further includes: The labels corresponding to the target feature data in the original image data of the current frame are enabled.

7. The image feature data classification method according to claim 6, characterized in that: The data packets corresponding to each set of target feature data are read from the target storage space, and each data packet is parsed to obtain each type of target feature data corresponding to the current frame original image, and each type of target data feature is sequentially transmitted to the back-end server according to a preset transmission method, including: Reading data packets corresponding to each set of target characteristic data from the target storage space according to the enable flag, parsing each data packet in real time, and finding the position of the end code to determine whether the target characteristic data of the current characteristic type has been read; If so, the current target feature data is transmitted to the backend server according to the transmission method corresponding to the target feature data of the current feature type.

8. The image feature data classification method according to claim 1, characterized in that: The feature extraction of the original image data of the current frame to obtain initial target feature data and background non-feature data includes: Edge detection processing is performed on the original image data of the current frame to obtain initial target feature data and background non-feature data corresponding to the original image data of the current frame, and feature extraction is performed on the initial target feature data to determine basic information of the original image data of the current frame, wherein the basic information includes: coordinate information and center line information.

9. The image feature data classification method according to any one of claims 1 to 8, characterized in that: The method further comprises: Before receiving the next frame of original image data, the auxiliary information of the current frame of original image data is deleted, wherein the auxiliary information includes: the label, the number of cycles, the characteristic information and the starting address of each target characteristic data in the current frame of original image.

10. An electronic device, characterized in that: The electronic device includes a field programmable gate array (FPGA), and the FPGA is used to execute the steps of the image feature data classification method according to any one of claims 1 to 9.

Images