Visual feature extraction method, device and electronic equipment

Abstract

The application discloses a visual feature extraction method and device and electronic equipment. The method comprises: for each frame image in a target image set, at a first starting time, performing image preprocessing on an Nth frame image in the target image set; the target image set is a sequence of image frames to be extracted visual features, the image preprocessing comprises a downsampling operation, and N is a positive integer greater than or equal to 1; at a second starting time, performing a visual feature extraction operation on the Nth frame image, and simultaneously performing image preprocessing on an N+1th frame image until each frame image completes the image preprocessing and the visual feature extraction operation, and the time point of the first starting time is before the second starting time; and outputting the visual features of each frame image in the target image set. The application solves the technical problem that a long time is consumed in extracting visual features in the related art.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a visual feature extraction method, apparatus, and electronic device. Background Technology

[0002] In compact visual search, visual feature extraction is the most complex part. Extracting scale-invariant feature transform (SIFT) features from a video graphics array (VGA) image typically takes several seconds on a regular processor and increases processor power consumption. As image resolution increases, the time required for visual feature extraction becomes longer, which cannot meet the low-latency response requirements of smart terminals. Summary of the Invention

[0003] This invention provides a visual feature extraction method, apparatus, and electronic device to at least solve the technical problem of long extraction time for visual features in related technologies.

[0004] According to one aspect of the present invention, a visual feature extraction method is provided, comprising: for each frame image in a target image set, at a first starting time, performing image preprocessing on the Nth frame image in the target image set; wherein the target image set is a sequence of image frames from which visual features are to be extracted, the image preprocessing includes a downsampling operation, and N is a positive integer greater than or equal to 1; performing visual feature extraction on the Nth frame image at a second starting time, and simultaneously performing image preprocessing on the (N+1)th frame image, until each frame image has completed image preprocessing and visual feature extraction, wherein the time point of the first starting time is before the second starting time; and outputting the visual features of each frame image in the target image set.

[0005] According to another aspect of the present invention, a visual feature extraction apparatus is also provided, comprising: a first processing unit, configured to perform image preprocessing on the Nth frame image in the target image set at a first starting time for each frame image in the target image set; wherein the target image set is a sequence of image frames from which visual features are to be extracted, the image preprocessing includes a downsampling operation, and N is a positive integer greater than or equal to 1; a second processing unit, configured to perform visual feature extraction on the Nth frame image at a second starting time, and simultaneously perform image preprocessing on the (N+1)th frame image, until each frame image has completed image preprocessing and visual feature extraction, wherein the first starting time is prior to the second starting time; and an output unit, configured to output the visual features of each frame image in the target image set.

[0006] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the above-described visual feature extraction method through the computer program.

[0007] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program is configured to execute the above-described visual feature extraction method at runtime.

[0008] In this embodiment of the invention, for each frame in the target image set, image preprocessing is performed on the Nth frame in the target image set at a first starting time; wherein the target image set is a sequence of image frames from which visual features are to be extracted, the image preprocessing includes downsampling, and N is a positive integer greater than or equal to 1; at a second starting time, visual feature extraction is performed on the Nth frame, and image preprocessing is performed on the (N+1)th frame simultaneously, until each frame has completed image preprocessing and visual feature extraction, and the first starting time is before the second starting time; the visual features of each frame in the target image set are output. In the above method, since multiple image processing operations for extracting visual features are performed simultaneously in a pipeline manner, it can not only shorten the time for extracting visual features of images, but also improve the efficiency of visual feature extraction, thereby solving the technical problem of long extraction time for visual features in related technologies. Attached Figure Description

[0009] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0010] Figure 1 This is a schematic diagram of the application environment of an optional visual feature extraction method according to an embodiment of the present invention;

[0011] Figure 2 This is a schematic diagram of an optional visual feature extraction process according to an embodiment of the present invention;

[0012] Figure 3 This is a schematic diagram illustrating the feature extraction complexity analysis of another optional visual feature extraction method according to an embodiment of the present invention;

[0013] Figure 4 This is a schematic diagram of the hardware and software architecture of another optional visual feature extraction method according to an embodiment of the present invention;

[0014] Figure 5This is a frame-level pipeline diagram of another optional visual feature extraction method according to an embodiment of the present invention;

[0015] Figure 6 This is a block-level pipeline diagram of another optional visual feature extraction method according to an embodiment of the present invention;

[0016] Figure 7 This is a schematic diagram of the feature detection structure framework of another optional visual feature extraction method according to an embodiment of the present invention;

[0017] Figure 8 This is a schematic diagram of the feature description structure framework of another optional visual feature extraction method according to an embodiment of the present invention;

[0018] Figure 9 This is a schematic diagram of the structure of an optional visual feature extraction device according to an embodiment of the present invention;

[0019] Figure 10 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] According to one aspect of the present invention, a visual feature extraction method is provided. Optionally, as an alternative implementation, the above-described visual feature extraction method may be applied to, but is not limited to, applications such as... Figure 1 The application environment shown. For example... Figure 1As shown, user 102 and user device 104 can interact with each other. User device 104 includes memory 106 and processor 108. In this embodiment, user device 104 may, but is not limited to, perform the following operations to obtain the visual features of each frame of image:

[0023] For each frame in the target image set, at the first starting time, the Nth frame in the target image set is preprocessed; wherein, the target image set is a sequence of image frames from which visual features are to be extracted, the image preprocessing includes a downsampling operation, and N is a positive integer greater than or equal to 1;

[0024] At the second starting moment, visual feature extraction is performed on the Nth frame image, and image preprocessing is performed on the N+1th frame image simultaneously, until image preprocessing and visual feature extraction are completed for each frame image. The time point of the first starting moment is before the second starting moment.

[0025] Output the visual features of each frame in the target image set mentioned above.

[0026] Optionally, the user equipment 104 mentioned above includes, but is not limited to, terminals such as mobile phones, tablets, laptops, PCs, in-vehicle electronic devices, and wearable devices. The above is merely an example, and no limitation is made in this embodiment.

[0027] like Figure 3 As shown, the complexity (computation time) of each algorithm module in the compact visual feature extraction of the Compact Descriptor for Visual Search (CDVS) ​​is as follows: Figure 3 As shown. CDVS is a standard technique for image visual features, from Figure 3 As can be seen, image preprocessing, feature point detection, and feature point description account for over 90% of the overall complexity. These three modules also involve numerous logical operations, making them suitable for hardware acceleration. Modules such as local feature aggregation and coordinate compression, on the other hand, have lower algorithmic complexity and are better suited for software implementation. Therefore, as... Figure 4 As shown, the hardware-implemented steps of this embodiment are image preprocessing, feature point detection, and feature point description; the software-implemented steps are image feature aggregation, image feature compression, and coordinate compression in the image.

[0028] As an alternative implementation method, such as Figure 2 As shown, this embodiment of the invention provides a visual feature extraction method, including the following steps:

[0029] S202, for each frame of the target image set, at the first starting time, the Nth frame of the target image set is preprocessed; wherein, the target image set is a sequence of image frames from which visual features are to be extracted, the image preprocessing includes downsampling operation, and N is a positive integer greater than or equal to 1;

[0030] S204, at the second starting time, the visual feature extraction operation is performed on the Nth frame image, and the image preprocessing is performed on the N+1th frame image simultaneously, until the image preprocessing and visual feature extraction operation are completed for each frame image, and the time point of the first starting time is before the second starting time.

[0031] S206, Output the visual features of each frame in the above target image set.

[0032] In embodiments of the present invention, such as Figure 5 As shown, Figure 5 The diagram illustrates a pipelined process that performs parallel feature extraction and preprocessing on each frame of a target image set. Here, image preprocessing includes, but is not limited to, downsampling each frame, which simultaneously scales the image width and height to 1 / N of their original values. For example, scaling the width and height of a 1920x1080 image to 1 / 4 of its original value results in a preprocessed image with a resolution of (1920 / 4)*(1080 / 4). Simultaneously, feature processing and preprocessing can be performed on different frames, shortening the processing time for visual feature extraction.

[0033] In this embodiment of the invention, for each frame in the target image set, image preprocessing is performed on the Nth frame in the target image set at a first starting time; wherein the target image set is a sequence of image frames from which visual features are to be extracted, the image preprocessing includes downsampling, and N is a positive integer greater than or equal to 1; at a second starting time, visual feature extraction is performed on the Nth frame, and image preprocessing is performed on the (N+1)th frame simultaneously, until each frame has completed image preprocessing and visual feature extraction, and the first starting time is before the second starting time; the visual features of each frame in the target image set are output. In the above method, since multiple image processing operations for extracting visual features are performed simultaneously in a pipeline manner, it can not only shorten the time for extracting visual features of images, but also improve the efficiency of visual feature extraction, thereby solving the technical problem of long extraction time for visual features in related technologies.

[0034] In one or more embodiments, the above visual feature extraction operation includes feature point detection and feature point description, and the above method further includes:

[0035] The following operations are performed sequentially on each frame of the target image set until each coding unit in each frame has completed feature point detection and feature point description:

[0036] Feature point detection is performed on the Nth coding unit at the third starting time.

[0037] At the fourth starting time, the Nth encoding unit is described by feature points, and the (N+1)th encoding unit is detected by feature points simultaneously; wherein, N is a positive integer greater than or equal to 1, and the time point of the third starting time is before the fourth starting time.

[0038] In embodiments of the present invention, such as Figure 6 As shown, Figure 6 The diagram illustrates the pipelined processing of each frame in the target image set at the block (coding unit) level. Simultaneously, feature point detection and feature point description can be performed on different frames in parallel, while feature processing and preprocessing are performed on different frames, thus shortening the processing time for visual feature extraction.

[0039] In one or more embodiments, the feature point detection described above includes key point detection and key point fine localization, and the method further includes:

[0040] The following operations are performed sequentially on each pixel in the current encoding unit until keypoint detection and keypoint fine localization are completed for each pixel in each encoding unit:

[0041] At the fifth starting moment, perform keypoint detection on the Nth pixel;

[0042] At the sixth starting moment, the Nth pixel is precisely located, and the (N+1)th pixel is simultaneously detected; N is a positive integer greater than or equal to 1, and the fifth starting moment is before the sixth starting moment.

[0043] In this embodiment of the invention, by means of the above-mentioned technical means, at the same time, feature processing and preprocessing of images of different frames can be performed in parallel, while feature point detection and feature point description of images of different frames can be performed in parallel, and key point detection and key point fine localization of each pixel in the current coding unit can be performed in parallel, thereby shortening the processing time of visual feature extraction.

[0044] In one or more embodiments, the visual feature extraction method further includes:

[0045] Based on the preset hardware processing module, image preprocessing and visual feature extraction operations are performed on each frame of the target image set.

[0046] Based on the software processing module associated with the hardware processing module, feature soft processing is performed on each frame of the target image set. The feature soft processing includes feature aggregation, feature compression, and coordinate compression.

[0047] like Figure 3 and Figure 4 As shown, image scaling (in the CDVS workflow, scaling, i.e., image preprocessing, is required first for high-resolution images), keypoint detection, and keypoint description account for over 90% of the overall complexity. Furthermore, the logical algorithms of these three modules are relatively regular, and hardware computation can improve the computational speed of these processes. In contrast, modules such as local feature aggregation and coordinate compression have lower algorithmic complexity but less regular operations; therefore, software computation can improve the computational speed of these processes.

[0048] In one or more embodiments, the above-described visual feature extraction method further includes:

[0049] Obtain the target Gaussian image from external storage. The target Gaussian image is the image used in the feature point description process for key pixels with similar spatial locations.

[0050] The target Gaussian image is stored in the on-chip memory so that key pixels with similar spatial locations can share the target Gaussian image.

[0051] In one or more embodiments, the visual features of each frame of an image are scale-invariant feature-transformed SIFT features.

[0052] Based on the above embodiments, in one application embodiment, the above visual feature extraction method combines frame-level and block-level pipelines to achieve hardware (processor) acceleration in the image extraction process, wherein the frame-level pipeline is as follows: Figure 5 As shown, when the first frame is preprocessed, the zeroth image can be extracted simultaneously.

[0053] Block-level pipelines such as Figure 6 As shown, feature extraction achieves two processes: feature detection and feature description. While the 0th coding unit in the current video frame performs feature description, the 1st coding unit can simultaneously perform feature detection. Generally, the more pipeline levels, the higher the data throughput and processing performance. This embodiment of the invention, based on actual computing and storage resource constraints, rationally divides the image into blocks (multiple coding units), thereby further saving computing resources.

[0054] like Figure 7 As shown, Figure 7The microstructure implementation block diagram for feature point detection and fine localization mainly consists of five parts: input image group buffer, control unit, keypoint detection, keypoint fine localization, and keypoint output. Keypoint detection is responsible for detecting key local feature points on the image from which visual features are to be extracted. Keypoint localization, based on keypoint detection, further refines the localization around the initially detected keypoints, finding their more precise locations. Simultaneously, edge points are filtered out based on the characteristics of the keypoint responses. Keypoint output integrates and outputs the remaining keypoint information after filtering. Keypoint detection and fine localization are implemented using a pixel-level pipeline.

[0055] Keypoint detection employs a low-order polynomial, associated Legendre polynomial (ALP), interest point detection method based on the CDVS standard. This method approximates the image through polynomial filtering, thereby detecting interest points. Furthermore, to reduce the memory required for interest point detection, CDVS also provides a frequency domain filtering method based on block-based scale space. This method divides the original scale space into several overlapping sub-blocks and then performs interest point detection on each sub-block. This block-based approach to interest point detection makes feature extraction more suitable for parallel acceleration and hardware implementation.

[0056] After keypoint detection, to accommodate different descriptor length constraints, the local feature set needs to be selected. Local features are ranked according to importance to obtain feature subsets. The CDVS standard feature selection first calculates a relevance score for each local feature, and then ranks the local features based on the calculated relevance. This relevance represents the prior probability that a local feature point of a query image is correctly matched by database features. The relevance score of a local feature is determined by information such as the keypoint's scale, scale-space response extrema, and the distance of the keypoint from the image center.

[0057] Feature point description microstructure diagram as follows Figure 8 As shown, the module mainly includes control logic, a data reading module, gradient calculation, local feature calculation, and feature summarization. The control logic is responsible for controlling the entire module and coordinating the normal operation of each part of the pipeline. The data reading module reads the Gaussian images required for keypoints with similar spatial locations from external memory into the on-chip memory, enabling multiple neighboring keypoints to share the Gaussian image, thus significantly saving bandwidth in the description module. Gradient calculation and local feature description of other features can be implemented using multi-path parallelism to improve hardware implementation speed.

[0058] During feature point description, for each detected keypoint, local features need to be described from the surrounding local region. The region surrounding the keypoint should be centered on the keypoint's location and rotated according to the keypoint's principal direction so that its horizontal axis aligns with the keypoint's principal direction. The local region of the keypoint must be divided into 4x4 (16) subspaces, each subspace being a unit. For each unit, each pixel is assigned to one of eight defined directions according to its gradient direction, resulting in an 8-dimensional histogram. A local region's gradient direction histogram is formed by sequentially concatenating the gradient direction histograms of these units. This forms the local feature descriptor, represented as a 128-dimensional histogram vector.

[0059] In this embodiment of the invention, local feature compression is achieved using a low-complexity transform coding scheme via CDVS. Local features are encoded through transform, quantization, and variable-length coding. This scheme transforms and encodes the eight directional vectors within the histogram of each SIFT unit. After transform and quantization, only elements in certain dimensions of the descriptor are selected and encoded into the bitstream. At different bitrates, the selected elements are determined according to a predefined standard table to maximize retrieval performance. The number of selected elements ranges from 20 to 128, determined by the descriptor length constraint.

[0060] For local feature aggregation and coordinate compression implemented through software programs, a position histogram encoding scheme is used for local location point compression. The position information of local feature points is converted into a statistical histogram through quantization statistics. The histogram encoding consists of two parts: label mapping encoding and label statistical graph encoding. The label statistical graph represents the number of points contained in the block containing feature points from top to bottom and left to right, while the label mapping represents the label matrix indicating whether local features exist in each grid. Finally, context-based arithmetic encoding is used to encode the histogram.

[0061] CDVS employs Scalable Compressed Fisher Vector (SCFV) for global feature aggregation. To compress the high-dimensional Fisher Vector, several discriminative Gaussian components from the Gaussian Mixture Model are selected and retained. SCFV exhibits excellent matching performance while having significantly lower memory overhead than typical Fisher Vector compression methods based on PCA or product quantization. The feature aggregation descriptors in the CDVS standard achieve high retrieval performance while meeting the low memory and low computational complexity requirements of each stage in the feature extraction process.

[0062] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0063] According to another aspect of the present invention, a visual feature extraction apparatus for implementing the above-described visual feature extraction method is also provided. For example... Figure 9 As shown, the device includes:

[0064] The first processing unit 902 is configured to perform image preprocessing on the Nth frame image in the target image set at a first starting time for each frame image in the target image set; wherein the target image set is a sequence of image frames from which visual features are to be extracted, the image preprocessing includes a downsampling operation, and N is a positive integer greater than or equal to 1;

[0065] The second processing unit 904 is used to perform visual feature extraction on the Nth frame image at the second starting time, and simultaneously perform image preprocessing on the N+1th frame image until each frame image has completed image preprocessing and visual feature extraction, wherein the time point of the first starting time is before the second starting time.

[0066] The output unit 906 is used to output the visual features of each frame of the target image set.

[0067] In this embodiment of the invention, for each frame in the target image set, image preprocessing is performed on the Nth frame in the target image set at a first starting time; wherein the target image set is a sequence of image frames from which visual features are to be extracted, the image preprocessing includes downsampling, and N is a positive integer greater than or equal to 1; at a second starting time, visual feature extraction is performed on the Nth frame, and image preprocessing is performed on the (N+1)th frame simultaneously, until each frame has completed image preprocessing and visual feature extraction, and the first starting time is before the second starting time; the visual features of each frame in the target image set are output. In the above method, since multiple image processing operations for extracting visual features are performed simultaneously in a pipeline manner, it can not only shorten the time for extracting visual features of images, but also improve the efficiency of visual feature extraction, thereby solving the technical problem of long extraction time for visual features in related technologies.

[0068] In one or more embodiments, the above-described visual feature extraction device further includes:

[0069] The third processing unit is configured to sequentially perform the following operations on each frame of the target image set until each encoding unit in each frame of the image has completed feature point detection and feature point description:

[0070] Feature point detection is performed on the Nth coding unit at the third starting time.

[0071] At the fourth starting time, the Nth encoding unit is described by feature points, and the (N+1)th encoding unit is detected by feature points simultaneously; wherein, N is a positive integer greater than or equal to 1, and the time point of the first starting time is before the second starting time.

[0072] In one or more embodiments, the feature point detection includes key point detection and key point fine localization, and the visual feature extraction device further includes:

[0073] The fourth processing unit is used to sequentially perform the following operations on each pixel in the current encoding unit until keypoint detection and keypoint fine localization are completed for each pixel in each encoding unit:

[0074] At the fifth starting moment, perform keypoint detection on the Nth pixel;

[0075] At the sixth starting moment, the Nth pixel is precisely located, and the (N+1)th pixel is simultaneously detected; N is a positive integer greater than or equal to 1, and the fifth starting moment is before the sixth starting moment.

[0076] In one or more embodiments, the visual feature extraction device further includes:

[0077] The hardware processing unit is used to perform image preprocessing and visual feature extraction operations on each frame of the target image set based on a preset hardware processing module.

[0078] The software processing unit is configured to perform feature soft processing on each frame of the target image set based on the software processing module associated with the hardware processing module. The feature soft processing includes feature aggregation, feature compression, and coordinate compression.

[0079] In one or more embodiments, the visual feature extraction device further includes:

[0080] The acquisition unit is used to acquire a target Gaussian image from an external memory. The target Gaussian image is an image used in the process of describing feature points for key pixels that are spatially close.

[0081] A storage unit is used to store the target Gaussian image into an on-chip memory so that key pixels with similar spatial locations can share the target Gaussian image.

[0082] According to another aspect of the present invention, an electronic device for implementing the above-described visual feature extraction method is also provided. This electronic device may be... Figure 10 The terminal device or server shown. This embodiment uses this electronic device as an example for illustration. Figure 10 As shown, the electronic device includes a processor 1002 and a memory 1004. The processor 1002 stores a computer program and is configured to execute the steps in any of the above method embodiments via the computer program.

[0083] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0084] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0085] S1, for each frame of the target image set, at the first starting time, the Nth frame of the target image set is preprocessed; wherein, the target image set is a sequence of image frames from which visual features are to be extracted, the image preprocessing includes downsampling operation, and N is a positive integer greater than or equal to 1;

[0086] S2, at the second starting moment, the visual feature extraction operation is performed on the Nth frame image, and the image preprocessing is performed on the N+1th frame image simultaneously, until the image preprocessing and visual feature extraction operation are completed for each frame image, and the time point of the first starting moment is before the second starting moment;

[0087] S3, output the visual features of each frame in the target image set.

[0088] Alternatively, as those skilled in the art will understand, Figure 10 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 10 This does not limit the structure of the aforementioned electronic devices or electronic equipment. For example, electronic devices or electronic equipment may also include components that are more... Figure 10 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 10 The different configurations shown.

[0089] The memory 1004 can be used to store software programs and modules, such as the program instructions / modules corresponding to the visual feature extraction method and apparatus in this embodiment of the invention. The processor 1002 executes various functional applications and data processing, that is, implements the aforementioned visual feature extraction method. The memory 1004 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the processor 1002 may further include memory remotely located relative to the memory 1004, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 1004 may be used, but is not limited to, to store information such as video frames and image features. As an example, such as... Figure 10 As shown, the processor 1002 may include, but is not limited to, the first processing unit 902, the second processing unit 904, and the output unit 906 in the aforementioned visual feature extraction device. Furthermore, it may include, but is not limited to, other module units in the aforementioned visual feature extraction device, which will not be elaborated upon in this example.

[0090] Optionally, the transmission device 1006 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 1006 includes a Network Interface Controller (NIC), which can be connected to other network devices and routers via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 1006 is a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0091] In addition, the above-mentioned electronic device also includes: a display 1008 for displaying the above-mentioned visual features; and a connection bus 1010 for connecting the various module components in the above-mentioned electronic device.

[0092] In other embodiments, the aforementioned terminal device or server can be a node in a distributed system, wherein the distributed system can be a blockchain system, which is a distributed system formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer (P2P) network, and any form of computing device, such as a server, terminal, or other electronic device, can become a node in the blockchain system by joining this peer-to-peer network.

[0093] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the visual feature extraction method described above. The computer program is configured to execute the steps of any of the method embodiments described above during runtime.

[0094] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0095] S1, for each frame of the target image set, at the first starting time, the Nth frame of the target image set is preprocessed; wherein, the target image set is a sequence of image frames from which visual features are to be extracted, the image preprocessing includes downsampling operation, and N is a positive integer greater than or equal to 1;

[0096] S2, at the second starting moment, the visual feature extraction operation is performed on the Nth frame image, and the image preprocessing is performed on the N+1th frame image simultaneously, until the image preprocessing and visual feature extraction operation are completed for each frame image, and the time point of the first starting moment is before the second starting moment;

[0097] S3, output the visual features of each frame in the above target image set.

[0098] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0099] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0100] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0101] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A visual feature extraction method, characterized in that, include: For each frame in the target image set, at the first starting time, the Nth frame in the target image set is preprocessed; wherein, the target image set is a sequence of image frames from which visual features are to be extracted, the image preprocessing includes a downsampling operation, and N is a positive integer greater than or equal to 1; At the second starting time, visual feature extraction is performed on the Nth frame image, and image preprocessing is performed on the N+1th frame image simultaneously, until image preprocessing and visual feature extraction are completed for each frame image. The first starting time is before the second starting time. The visual feature extraction operation includes feature point detection and feature point description. Output the visual features of each frame in the target image set; The following operations are performed sequentially on each frame of the target image set until each coding unit in each frame has completed feature point detection and feature point description: At the third starting moment, feature point detection is performed on the Nth coding unit, and the feature point detection includes key point detection and key point fine localization. At the fourth starting time, the Nth encoding unit is described by feature points, and the (N+1)th encoding unit is detected by feature points simultaneously; wherein, N is a positive integer greater than or equal to 1, and the time point of the third starting time is before the fourth starting time; The following operations are performed sequentially on each pixel in the current encoding unit until keypoint detection and keypoint fine localization are completed for each pixel in each encoding unit: At the fifth starting moment, perform keypoint detection on the Nth pixel; At the sixth starting moment, the Nth pixel is precisely located, and the (N+1)th pixel is simultaneously detected; N is a positive integer greater than or equal to 1, and the fifth starting moment is before the sixth starting moment. Obtain the target Gaussian image from external storage. The target Gaussian image is an image used in the process of describing feature points for key pixels that are spatially close. The target Gaussian image is stored in on-chip memory so that key pixels with similar spatial locations can share the target Gaussian image.

2. The method according to claim 1, characterized in that, The visual features of each frame of the image are scale-invariant feature transform (SIFT) features.

3. The method according to claim 1, characterized in that, The method further includes: Based on the preset hardware processing module, image preprocessing and visual feature extraction operations are performed on each frame of the target image set. Based on the software processing module associated with the hardware processing module, feature soft processing is performed on each frame of the target image set. The feature soft processing includes feature aggregation, feature compression, and coordinate compression.

4. A visual feature extraction device, characterized in that, include: The first processing unit is configured to perform image preprocessing on the Nth frame image in the target image set at a first starting time for each frame image in the target image set; wherein the target image set is a sequence of image frames from which visual features are to be extracted, the image preprocessing includes a downsampling operation, and N is a positive integer greater than or equal to 1; The second processing unit is used to perform visual feature extraction on the Nth frame image at the second starting time, and simultaneously perform image preprocessing on the N+1th frame image, until each frame image has completed image preprocessing and visual feature extraction. The first starting time is before the second starting time. The visual feature extraction operation includes feature point detection and feature point description. The output unit is used to output the visual features of each frame of the target image set; The third processing unit is configured to sequentially perform the following operations on each frame of the target image set until each encoding unit in each frame of the image has completed feature point detection and feature point description: at the third starting time, feature point detection is performed on the Nth encoding unit, the feature point detection including key point detection and key point fine localization; at the fourth starting time, feature point description is performed on the Nth encoding unit, and feature point detection is performed on the (N+1)th encoding unit simultaneously; wherein, N is a positive integer greater than or equal to 1, and the time point of the third starting time is before the fourth starting time; The fourth processing unit is configured to sequentially perform the following operations on each pixel in the current encoding unit until keypoint detection and keypoint fine localization are completed for each pixel in each encoding unit: keypoint detection is performed on the Nth pixel at the fifth starting time; keypoint fine localization is performed on the Nth pixel at the sixth starting time, and keypoint detection is performed on the (N+1)th pixel simultaneously; where N is a positive integer greater than or equal to 1, and the time point of the fifth starting time is before the sixth starting time; The acquisition unit is used to acquire a target Gaussian image from an external memory. The target Gaussian image is an image used in the process of describing feature points for key pixels that are spatially close. A storage unit is used to store the target Gaussian image into an on-chip memory so that key pixels with similar spatial locations can share the target Gaussian image.

5. An electronic device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 3 through the computer program.

Images