An object recognition method, apparatus, device, and storage medium

Abstract

This application discloses an object recognition method, apparatus, device, and storage medium. The method includes: acquiring first sensing information of an object to be recognized, wherein the first sensing information is information of a terahertz electromagnetic wave reflected by the object to be recognized; determining the time-spectrum map corresponding to the first sensing information; inputting the time-spectrum map into a pre-trained object recognition model; determining the classification information corresponding to the time-spectrum map; and determining the type information of the object to be recognized based on the classification information. In this way, the limitation of traditional radar in being unable to classify and recognize objects can be avoided, ensuring the accuracy and efficiency of object recognition and detection, while reducing the dependence on visual sensors.

Description

Technical Field

[0001] This application relates to the field of radar detection technology, and in particular to an object recognition method, device, equipment and storage medium. Background Technology

[0002] With the development of autonomous driving technology, autonomous vehicles are playing an increasingly important role in the automotive industry, and the requirements for the real-time performance, accuracy, and efficiency of environmental perception and object recognition in autonomous driving systems are also becoming increasingly stringent. Existing object recognition and detection methods are mainly divided into two categories: the first is based on computer vision technology, which uses features such as color and shape for edge detection, image segmentation, and target recognition; the second is based on radar sensors, which calculates the location and distance of objects by analyzing the time difference and intensity changes between transmitted and received waves. However, computer vision-based object recognition and detection methods are limited by the computing power of hardware and the performance of software algorithms in terms of both real-time performance and accuracy. While radar sensor-based object recognition and detection methods offer better real-time performance and accuracy compared to computer vision-based methods, they are lacking in object recognition capabilities. Therefore, a new object recognition and detection method is needed that can solve the problems of real-time performance and accuracy in object recognition. Summary of the Invention

[0003] This application provides an object recognition method, apparatus, device, and storage medium that can circumvent the limitations of traditional radar in classifying and recognizing objects, ensure the accuracy and efficiency of object recognition and detection, and reduce reliance on visual sensors.

[0004] On one hand, embodiments of this application provide an object recognition method, the method comprising:

[0005] Acquire the first sensing information of the object to be identified; the first sensing information is the information of the terahertz electromagnetic wave after being reflected by the object to be identified;

[0006] Determine the time-spectrum diagram corresponding to the first sensing information;

[0007] Input the time-spectrum image into the trained object recognition model to determine the classification information corresponding to the time-spectrum image;

[0008] The classification information determines the type of object to be identified.

[0009] Further, determining the time-spectrum diagram corresponding to the first sensing information includes:

[0010] The first sensing information is denoised to obtain the denoised first sensing information.

[0011] The noise-reduced first sensor information is converted into time-domain information;

[0012] The time-domain spectrogram is determined based on time-domain information.

[0013] Furthermore, the first sensing information is denoised to obtain denoised first sensing information, including:

[0014] Wavelet decomposition is performed on the first sensing information to obtain the scaling coefficient of the first sensing information;

[0015] The scaling coefficients are thresholded to obtain the thresholded scaling coefficients.

[0016] Wavelet reconstruction is performed on the scale coefficients after thresholding to obtain the first sensing information after denoising.

[0017] Furthermore, determining the time-spectrum based on time-domain information includes:

[0018] The first time-domain spectrum is obtained by performing a short-time Fourier transform on the time-domain information.

[0019] and / or;

[0020] Wavelet transform is performed on the time-domain information to obtain the second time-spectrum.

[0021] Furthermore, the time-spectrum image is input into the trained object recognition model to determine the classification information corresponding to the time-spectrum image, including:

[0022] Input the first temporal spectrum into the pre-trained first object recognition model to determine the first classification information corresponding to the first temporal spectrum;

[0023] and / or;

[0024] The second time-spectrum image is input into the pre-trained second object recognition model to determine the second classification information corresponding to the second time-spectrum image.

[0025] Furthermore, the category information of the object to be identified is determined based on the classification information, including:

[0026] The category information of the object to be identified is determined based on the first category information and / or the second category information.

[0027] Furthermore, the object recognition method also includes the step of training a first object recognition model:

[0028] The first object recognition model obtained through training includes:

[0029] Obtain the first sample dataset, which includes the first time-frequency spectrum of each sample object and the historical classification information of each sample object;

[0030] Construct a first preset machine learning model and determine the first preset machine learning model as the first current machine learning model;

[0031] Based on the first current machine learning model, classification information is predicted for the first time spectrum map to determine the predicted classification information corresponding to the first time spectrum map;

[0032] Based on the predicted classification information and historical classification information corresponding to the first time spectrum, the loss value is determined;

[0033] When the loss value is greater than a preset threshold, backpropagation is performed based on the loss value to update the first current machine learning model to obtain an updated machine learning model. The updated machine learning model is then re-determined as the first current machine learning model. The following steps are repeated: Based on the first current machine learning model, classification information is predicted for the first time-spectrum map to determine the predicted object class information corresponding to the first time-spectrum map.

[0034] When the loss value is less than or equal to a preset threshold, the first current machine learning model is determined as the first object recognition model.

[0035] Furthermore, the object recognition method also includes the step of training a second object recognition model:

[0036] The trained second object recognition model includes:

[0037] Obtain the second sample dataset, which includes the second time-frequency spectrum corresponding to each sample object in multiple sample objects and the historical classification information corresponding to each sample object;

[0038] Construct a second preset machine learning model and determine the second preset machine learning model as the second current machine learning model;

[0039] Based on the second current machine learning model, classification information is predicted for the second time-spectrum map to determine the predicted classification information corresponding to the second time-spectrum map;

[0040] Based on the predicted classification information and historical classification information corresponding to the second time-spectrum map, the loss value is determined;

[0041] When the loss value is greater than a preset threshold, backpropagation is performed based on the loss value to update the second current machine learning model to obtain an updated machine learning model. The updated machine learning model is then re-determined as the second current machine learning model. The following steps are repeated: Based on the second current machine learning model, classification information is predicted for the second time-spectrum map to determine the predicted object class information corresponding to the second time-spectrum map.

[0042] When the loss value is less than or equal to a preset threshold, the second current machine learning model is determined as the second object recognition model.

[0043] Furthermore, object recognition methods also include:

[0044] Acquire the second sensing information of the object to be identified; the second sensing information is the information of the radar signal after being reflected by the object to be identified;

[0045] Based on the second sensor information, determine the distance information and instance information between the object to be identified and the current position;

[0046] The objects to be identified are fused based on distance information, instance information, and category information.

[0047] On the other hand, embodiments of this application provide an object recognition device, which includes:

[0048] The sensor information acquisition module is used to acquire the first sensor information of the object to be identified; the first sensor information is the information of the terahertz electromagnetic wave after being reflected by the object to be identified.

[0049] The first determining module is used to determine the time spectrum corresponding to the first sensing information;

[0050] The second determination module is used to input the time spectrum into the trained object recognition model and determine the classification information corresponding to the time spectrum.

[0051] The third determination module is used to determine the type of the object to be identified based on the classification information.

[0052] Furthermore, the first determining module is used to perform denoising processing on the first sensing information to obtain denoised first sensing information; convert the denoised first sensing information into time domain information; and determine the time spectrum based on the time domain information.

[0053] Furthermore, the first determining module is used to perform wavelet decomposition on the first sensing information to obtain the scaling coefficients of the first sensing information; to perform threshold processing on the scaling coefficients to obtain the threshold-processed scaling coefficients; and to perform wavelet reconstruction on the threshold-processed scaling coefficients to obtain the denoised first sensing information.

[0054] Furthermore, the first determining module is also used to perform a short-time Fourier transform on the time-domain information to obtain a first time-spectrum; and / or to perform a wavelet transform on the time-domain information to obtain a second time-spectrum.

[0055] Furthermore, the second determining module is used to input the first temporal spectrogram into the pre-trained first object recognition model to determine the first classification information corresponding to the first temporal spectrogram; and / or to input the second temporal spectrogram into the pre-trained second object recognition model to determine the second classification information corresponding to the second temporal spectrogram.

[0056] Furthermore, the third determining module is used to determine the type information of the object to be identified based on the first classification information and / or the second classification information.

[0057] Furthermore, the second determining module is also used to acquire a first sample dataset, which includes a first time-frequency spectrum corresponding to each sample object among multiple sample objects and historical classification information corresponding to each sample object; construct a first preset machine learning model and determine the first preset machine learning model as the first current machine learning model; predict classification information for the first time-frequency spectrum based on the first current machine learning model to determine the predicted classification information corresponding to the first time-frequency spectrum; determine a loss value based on the predicted classification information and historical classification information corresponding to the first time-frequency spectrum; when the loss value is greater than a preset threshold, backpropagation is performed based on the loss value to update the first current machine learning model to obtain an updated machine learning model, and the updated machine learning model is re-determined as the first current machine learning model; repeat the steps: predict classification information for the first time-frequency spectrum based on the first current machine learning model to determine the predicted object class information corresponding to the first time-frequency spectrum; when the loss value is less than or equal to the preset threshold, the first current machine learning model is determined as the first object recognition model.

[0058] Furthermore, the second determining module is also used to acquire a second sample dataset, which includes a second time-frequency spectrum corresponding to each sample object among multiple sample objects and historical classification information corresponding to each sample object; construct a second preset machine learning model and determine the second preset machine learning model as the second current machine learning model; based on the second current machine learning model, perform classification information prediction on the second time-frequency spectrum to determine the predicted classification information corresponding to the second time-frequency spectrum; determine the loss value based on the predicted classification information and historical classification information corresponding to the second time-frequency spectrum; when the loss value is greater than a preset threshold, backpropagation is performed based on the loss value to update the second current machine learning model to obtain an updated machine learning model, and the updated machine learning model is re-determined as the second current machine learning model; repeat the steps: based on the second current machine learning model, perform classification information prediction on the second time-frequency spectrum to determine the predicted object class information corresponding to the second time-frequency spectrum; when the loss value is less than or equal to the preset threshold, determine the second current machine learning model as the second object recognition model.

[0059] Furthermore, the sensor information acquisition module is also used to acquire second sensor information of the object to be identified; the second sensor information is the information of the radar signal after being reflected by the object to be identified.

[0060] Furthermore, the device also includes:

[0061] The fourth determining module is used to determine the distance information and instance information between the object to be identified and the current position based on the second sensing information;

[0062] The fusion processing module is used to perform fusion processing on the object to be identified based on distance information, instance information, and category information.

[0063] On the other hand, embodiments of this application provide an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded by the processor and executed as described above for the object recognition method.

[0064] On the other hand, embodiments of this application provide a computer storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the object recognition method as described above.

[0065] The object recognition method, apparatus, device, and storage medium provided in this application have the following technical effects:

[0066] The first sensing information of the object to be identified is obtained. The first sensing information is the information of the terahertz electromagnetic wave reflected by the object. The time spectrum corresponding to the first sensing information is determined. The time spectrum is input into the pre-trained object recognition model to determine the classification information corresponding to the time spectrum. Based on the classification information, the type of the object to be identified is determined. In this way, the limitation of traditional radar in being unable to identify and classify objects can be avoided, ensuring the accuracy and efficiency of object recognition and detection, while reducing the dependence on visual sensors. Attached Figure Description

[0067] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application;

[0069] Figure 2 This is a flowchart illustrating an object recognition method provided in an embodiment of this application;

[0070] Figure 3 This is a flowchart illustrating the determination of the time-spectrum diagram corresponding to the first sensing information in an object recognition method provided in this application embodiment;

[0071] Figure 4 This application provides a method for training a first object recognition model.

[0072] Figure 5This application provides a method for training a second object recognition model.

[0073] Figure 6 This is a structural diagram of an object recognition model provided in an embodiment of this application;

[0074] Figure 7 This is a flowchart illustrating an object recognition method provided in an embodiment of this application;

[0075] Figure 8 This is a schematic diagram of the structure of an object recognition device provided in an embodiment of this application;

[0076] Figure 9 This is a hardware structure block diagram of a server for an object recognition method provided in an embodiment of this application. Detailed Implementation

[0077] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0078] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0079] Please see Figure 1 , Figure 1This is a schematic diagram of an application environment provided in an embodiment of this application, including a sensor information acquisition module 101, a first determination module 102, a second determination module 103, and a third determination module 104. The sensor information acquisition module 101 acquires sensor information obtained after terahertz electromagnetic waves are reflected by an object to be identified. The first determination module 102 determines the time-spectrum diagram corresponding to the sensor information. The second determination module 103 inputs the time-spectrum diagram into a trained object recognition model to determine the classification information corresponding to the time-spectrum diagram. The third determination module 104 determines the type information of the object to be identified based on the classification information. After acquiring the first sensor information, the sensor information acquisition module 101 outputs the type information of the object to be identified sequentially through the first determination module 102, the second determination module 103, and the third determination module 104.

[0080] In this embodiment, the sensor information acquisition module 101, the first determination module 102, the second determination module 103, and the third determination module 104 can be housed in the same device, such as a mobile terminal, a computer terminal, a server, or a similar computing device. Optionally, the sensor information acquisition module 101, the first determination module 102, the second determination module 103, and the third determination module 104 can be housed in multiple devices, which are located in a single system. Optionally, the sensor information acquisition module 101, the first determination module 102, the second determination module 103, and the third determination module can be housed on a single platform. The execution subject of this embodiment is not limited to the control system of a traditional automobile or an autonomous vehicle, but can also be a mobile terminal, a computer terminal, a server, or a similar computing device; it can be a system or a platform.

[0081] In this embodiment of the application, the sensor information acquisition module 101, the first determination module 102, the second determination module 103 and the third determination module 104 can be connected by wire or wireless means.

[0082] The following describes a specific embodiment of an object recognition method according to this application. Figure 2 This is a flowchart illustrating an object recognition method provided in an embodiment of this application. This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or server products, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 2 As shown, the method may include:

[0083] S201: Obtain the first sensor information of the object to be identified.

[0084] In this embodiment, the first sensing information is the information obtained after terahertz electromagnetic waves are reflected by the object to be identified. After the terahertz electromagnetic waves are emitted by the transmitter of the terahertz sensor to the surface of the object to be identified, part of them are absorbed by the object and the other part is reflected by the object and received by the receiver of the terahertz sensor to obtain the first sensing information.

[0085] S203: Determine the time spectrum corresponding to the first sensing information.

[0086] In this embodiment, the first sensing information is frequency domain information. To obtain the time-spectrum diagram corresponding to the first sensing information, it is necessary to first convert the first sensing information into time domain information. Specifically, Figure 3 This illustration shows a flowchart of a method for determining the time-spectrum map corresponding to the first sensing information in an embodiment of this application. Figure 3 As shown, determining the time-spectrum diagram corresponding to the first sensing information may include:

[0087] S301: Denoise the first sensing information to obtain the denoised first sensing information.

[0088] In this embodiment, the first sensing information is denoised using wavelet decomposition and wavelet reconstruction. Specifically, wavelet decomposition is performed on the first sensing information to obtain its scaling coefficients, which include scaling coefficients of the effective signal and scaling coefficients of the noise. A suitable threshold is selected, and the scaling coefficients are thresholded. Scale coefficients smaller than the threshold are considered noise scaling coefficients and are set to zero, while scaling coefficients equal to or greater than the threshold are considered effective signal scaling coefficients and are retained, resulting in thresholded scaling coefficients. The thresholded scaling coefficients contain only the scaling coefficients of the effective signal. These scaling coefficients contain the effective information of the first sensing information, and wavelet reconstruction of them yields the denoised first sensing information.

[0089] S303: Converts the noise-reduced first sensor information into time-domain information.

[0090] In this embodiment, the denoised first sensing information is converted into time-domain information using Inverse Fast Fourier Transform (IFFT). Specifically, the first sensing information is transformed into a Cartesian coordinate system, its conjugate is taken, and then a Fast Fourier Transform (FFT) is performed. The transformed result is then conjugate again to obtain the time-domain information corresponding to the first sensing information.

[0091] S305: Determine the time-spectrum diagram based on time-domain information.

[0092] In this embodiment, the time-frequency spectrum includes a first time-frequency spectrum and a second time-frequency spectrum. The first time-frequency spectrum is obtained by performing a Short-Time Fourier Transform (STFT) on the time-domain information, and the second time-frequency spectrum is obtained by performing a wavelet transform on the time-domain information. Optionally, only the time-domain information can be subjected to a STFT to obtain the first time-frequency spectrum, and the type information of the object to be identified can be determined based on the first time-frequency spectrum. Optionally, only the time-domain information can be subjected to a wavelet transform to obtain the second time-frequency spectrum, and the type information of the object to be identified can be determined based on the second time-frequency spectrum. For systems or servers with limited hardware computing power and software algorithm performance, performing only a STFT or wavelet transform based on one time-frequency spectrum can accurately and efficiently determine the type information of the object to be identified. Optionally, both STFT and wavelet transform can be performed on the time-frequency information to obtain the first time-frequency spectrum and the second time-frequency spectrum, and the type information of the object to be identified can be determined by combining the first time-frequency spectrum and the second time-frequency spectrum.

[0093] S205: Input the time-spectrum image into the trained object recognition model to determine the classification information corresponding to the time-spectrum image.

[0094] In this embodiment of the application, for a first time-spectrum image, there is a corresponding first object recognition model. By inputting the first time-spectrum image into the first object recognition model, the first classification information corresponding to the first time-spectrum image can be determined. For a second time-spectrum image, there is a corresponding second object recognition model. By inputting the second time-spectrum image into the second object recognition model, the second classification information corresponding to the second time-spectrum image can be determined.

[0095] Optionally, for systems or servers that only perform Short Time Fourier Transform (STFT), only the first object recognition model may be included; for systems or servers that only perform Wavelet Transform, only the second object recognition model may be included.

[0096] S207: Determine the type of the object to be identified based on the classification information.

[0097] In this embodiment of the application, for a system or server that only includes a first object recognition model, the type information of the object to be recognized is determined only based on the first classification information; for a system or server that only includes a second object recognition model, the type information of the object to be recognized is determined only based on the second classification information.

[0098] As an optional implementation, for a system or server that has two object recognition models, a first object recognition model and a second object recognition model, the type information of the object to be recognized can be determined either based solely on the first classification information or the second classification information, or by combining the first classification information and the second classification information.

[0099] This application also provides a method for training a first object recognition model, such as... Figure 4 As shown, it includes:

[0100] S401: Obtain the first sample dataset, which includes the first time-frequency spectrum of each sample object and the historical classification information of each sample object.

[0101] In this embodiment, the sample articles in the sample dataset can be stored in a storage area, which can be a blockchain. A blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0102] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operational monitoring. The user management module is responsible for managing the identity information of all blockchain participants, including maintaining public and private key generation (account management), key management, and maintaining the correspondence between user real identities and blockchain addresses (access management). Furthermore, under authorization, it monitors and audits transactions of certain real identities and provides risk control rule configuration (risk control audit). The basic services module is deployed on all blockchain node devices to verify the validity of business requests. After consensus is reached on valid requests, they are recorded in storage. For a new business request, the basic services first perform interface adaptation parsing and authentication (interface adaptation), and then encrypt the business information through a consensus algorithm (consensus management). The blockchain process involves several layers of data transmission and processing. The first layer, the blockchain service layer, provides basic capabilities and implementation frameworks for typical applications. Developers can define contract logic using a programming language and publish it to the blockchain (contract registration). The second layer provides blockchain-based application services for business stakeholders. The third layer provides application services based on blockchain solutions for use by business participants.

[0103] S403: Construct a first preset machine learning model and determine the first preset machine learning model as the first current machine learning model.

[0104] S405: Based on the first current machine learning model, perform classification information prediction on the first time spectrum and determine the predicted classification information corresponding to the first time spectrum.

[0105] S407: Determine the loss value based on the predicted classification information and historical classification information corresponding to the first time spectrum.

[0106] S4011: If the loss value is greater than the preset threshold, proceed to step S411; otherwise, proceed to step S413.

[0107] S411: Perform backpropagation based on the loss value to update the first current machine learning model to obtain the updated machine learning model, and re-determine the updated machine learning model as the first current machine learning model; proceed to step S405.

[0108] S413: When the loss value is less than or equal to the preset threshold, the first current machine learning model is determined as the first object recognition model.

[0109] Based on the same method, embodiments of this application provide a training method for a second object recognition model, such as... Figure 5 As shown, it includes:

[0110] S501: Obtain the second sample dataset, which includes the second time-frequency spectrum of each sample object and the historical classification information of each sample object.

[0111] S503: Construct a second preset machine learning model and determine the second preset machine learning model as the second current machine learning model.

[0112] S505: Based on the second current machine learning model, perform classification information prediction on the second time-spectrum map to determine the predicted classification information corresponding to the second time-spectrum map.

[0113] S507: Determine the loss value based on the predicted classification information and historical classification information corresponding to the second time-spectrum map.

[0114] S5011: When the loss value is greater than the preset threshold, proceed to step S511; otherwise, proceed to step S513.

[0115] S511: Perform backpropagation based on the loss value to update the second current machine learning model to obtain the updated machine learning model, and redefine the updated machine learning model as the second current machine learning model; proceed to step S505.

[0116] S513: When the loss value is less than or equal to a preset threshold, the second current machine learning model is determined as the second object recognition model.

[0117] Figure 6 The diagram shows a structural diagram of an object recognition model provided in an embodiment of this application, as follows: Figure 6 As shown, the first object recognition model and the second object recognition model can be convolutional neural network models (CNN) based on the VGG16 system, including an input layer, multiple convolutional layers, multiple pooling layers, multiple fully connected layers and an output layer.

[0118] As an optional implementation, multiple convolutional layers, multiple pooling layers, and multiple fully connected layers are connected in series, with the convolutional layers and pooling layers spaced apart. Each convolutional layer may include two or three convolutional layers. Optionally, the multiple pooling layers may be max pooling layers.

[0119] As one alternative implementation, the object recognition model comprises, in sequence, an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, a fourth convolutional layer, a fourth pooling layer, a fifth convolutional layer, a fifth pooling layer, a first fully connected layer, a second fully connected layer, a third fully connected layer, and an output layer. The first convolutional layer may include a first convolutional layer and a second convolutional layer; the second convolutional layer may include a third convolutional layer and a fourth convolutional layer; the third convolutional layer may include a fifth convolutional layer, a sixth convolutional layer, and a seventh convolutional layer; the fourth convolutional layer may include an eighth convolutional layer, a ninth convolutional layer, and a tenth convolutional layer; and the fifth convolutional layer may include an eleventh convolutional layer, a twelfth convolutional layer, and a thirteenth convolutional layer. Optionally, the first fully connected layer can be replaced by a global pooling layer.

[0120] Taking a 224*224 time-spectrum image as an example, after the input layer receives the time-spectrum image, it is input into the first convolutional layer. The first convolutional layer includes 64 3*3 convolutional kernels, which perform convolution operations on it to obtain 64 224*224 time-spectrum images. The second convolutional layer receives these 64 224*224 time-spectrum images, and performs convolution operations on them using 64 3*3 convolutional kernels to obtain 64 224*224 time-spectrum images.

[0121] The first pooling layer receives the 64 224*224 time-spectrum images, performs max pooling through its pooling window, and then outputs 64 112*112 time-spectrum images. The first pooling layer includes a 2*2 pooling window.

[0122] The second convolutional layer receives the 64 112*112 time-frequency spectra. The third convolutional layer, consisting of 128 3*3 convolutional kernels, performs convolution operations on these 128 112*112 time-frequency spectra, resulting in another 128 112*112 time-frequency spectra. The fourth convolutional layer receives these 128 112*112 time-frequency spectra, and performs convolution operations on them using the same 128 3*3 convolutional kernels.

[0123] The second pooling layer receives the 128 112*112 time-spectrum images, performs max pooling through its pooling window, and then outputs 56 112*112 time-spectrum images. The second pooling layer includes a 2*2 pooling window.

[0124] The third convolutional layer receives the 56 112*112 time-frequency spectra. The fifth convolutional layer, consisting of 256 3*3 convolutional kernels, performs convolution operations on these 256 kernels to obtain 256 56*56 time-frequency spectra. The sixth convolutional layer receives the 256 56*56 time-frequency spectra, and performs convolution operations on them using the same kernels. The seventh convolutional layer receives the 256 56*56 time-frequency spectra, and performs convolution operations on them using the same kernels.

[0125] The third pooling layer receives the 256 56*56 time-spectrum images, performs max pooling through its pooling window, and then outputs 256 28*28 time-spectrum images. The third pooling layer includes a 2*2 pooling window.

[0126] The fourth convolutional layer receives the 256 28*28 time-spectrum maps. The eighth convolutional layer, consisting of 512 3*3 convolutional kernels, performs a convolution operation on these 512 28*28 time-spectrum maps to obtain 512 time-spectrum maps. The ninth convolutional layer receives the 512 28*28 time-spectrum maps and performs a convolution operation on them using 512 3*3 convolutional kernels to obtain 512 28*28 time-spectrum maps. The tenth convolutional layer receives the 512 28*28 time-spectrum maps and performs a convolution operation on them using 512 3*3 convolutional kernels to obtain 512 28*28 time-spectrum maps.

[0127] The fourth pooling layer receives the 512 28*28 time-spectrum images, performs max pooling through its pooling window, and then outputs 512 14*14 time-spectrum images. The fourth pooling layer includes a 2*2 pooling window.

[0128] The fifth convolutional layer receives the 512 14*14 time-frequency spectra. The eleventh convolutional layer, consisting of 512 3*3 convolutional kernels, performs a convolution operation on it to obtain 12 14*14 time-frequency spectra. The twelfth convolutional layer receives the 512 14*14 time-frequency spectra, and performs a convolution operation on it with the 512 3*3 convolutional kernels to obtain 512 14*14 time-frequency spectra. The thirteenth convolutional layer receives the 512 14*14 time-frequency spectra, and performs a convolution operation on it with the 512 3*3 convolutional kernels to obtain 512 14*14 time-frequency spectra.

[0129] The fifth pooling layer receives the 512 14*14 time-spectrum images, performs max pooling through its pooling window, and then outputs 512 7*7 time-spectrum images. The fourth pooling layer includes a 2*2 pooling window.

[0130] The first fully connected layer receives the 512 7*7 time spectrum maps, processes them, and outputs a 1*4096 time spectrum map.

[0131] The second fully connected layer receives the 1*4096 time spectrum, processes it, and then outputs the 1*4096 time spectrum.

[0132] The third fully connected layer receives the 1*4096 time spectrum, processes it, and outputs a 1*1000 time spectrum.

[0133] The output layer obtains a 1*1000 time-spectrum image. The output layer includes a Softmax classification module, which determines the degree of matching between the time-spectrum image and each classification information in the classification information set. The classification information corresponding to the maximum degree of matching is determined as the classification information of the object to be identified.

[0134] In this embodiment of the application, the structure of the object recognition model can be intuitively represented by a table. Table 1 is a correspondence table of the structure of an object recognition model according to an embodiment of the application. The first row of Table 1 represents the structure of the model, including the convolution kernel or pooling window contained in the layer and the output of the layer.

[0135] Table 1. Structure of an object recognition model

[0136]

[0137]

[0138] Figure 7 This document illustrates a flowchart of an object recognition method provided in an embodiment of this application. Figure 7 As shown, the object recognition method also includes:

[0139] S701: Acquire the second sensor information of the object to be identified.

[0140] In this embodiment, the second sensing information is the information obtained after the radar signal is reflected by the object to be identified. This radar signal is emitted by a traditional lidar sensor or radar sensor, and after being absorbed and reflected by the object to be identified, it is received by the receiver of the lidar sensor or radar sensor to obtain the second sensing information.

[0141] S703: Determine the distance information and instance information between the object to be identified and the current position based on the second sensor information.

[0142] In this embodiment, based on the second sensing information and the time difference and intensity change during reflection, distance information and instance information between the object to be identified and the current position are determined. The distance information determines the distance and orientation between the object to be identified and the current position, while the instance information determines the number of individuals included in the object to be identified. For example, if the object to be identified is a cyclist, the instance information can determine that the object to be identified includes two individuals.

[0143] S705: Perform fusion processing on the object to be identified based on distance information, instance information, and category information.

[0144] In this embodiment, distance information determines the distance and orientation between the object to be identified and the current location; instance information determines the number of individuals included in the object; and category information determines the type of each individual in the object, such as a person, a car, or a plant. Taking a cyclist 100 meters directly in front of the current location as an example, distance information determines that the cyclist is directly in front of the current location at a distance of 100 meters; instance information determines that the cyclist comprises two individuals; and category information determines that these two individuals are a person and a vehicle. By fusing these three types of information, specific information about the object to be identified can be obtained. This not only identifies the distance and orientation between the object and the current location but also the number and type of individuals contained in the object. This allows for efficient and accurate identification of the object, with low reliance on visual sensors and relatively low requirements for hardware computing power and software algorithm performance.

[0145] This application also provides an object recognition device. Figure 8 This is a schematic diagram of the structure of an object recognition device provided in an embodiment of this application, as shown below. Figure 8 As shown, the device includes:

[0146] The sensor information acquisition module 801 is used to acquire the first sensor information of the object to be identified; the first sensor information is the information of the terahertz electromagnetic wave after being reflected by the object to be identified.

[0147] The first determining module 802 is used to determine the time spectrum corresponding to the first sensing information.

[0148] The second determining module 803 is used to input the time spectrum into the trained object recognition model and determine the classification information corresponding to the time spectrum.

[0149] The third determination module 804 is used to determine the type information of the object to be identified based on the classification information.

[0150] As an optional implementation, the first determining module 802 is used to perform denoising processing on the first sensing information to obtain denoised first sensing information; convert the denoised first sensing information into time domain information; and determine the time spectrum based on the time domain information.

[0151] As an optional implementation, the first determining module 802 is used to perform wavelet decomposition on the first sensing information to obtain the scale coefficients of the first sensing information; to perform threshold processing on the scale coefficients to obtain the threshold-processed scale coefficients; and to perform wavelet reconstruction on the threshold-processed scale coefficients to obtain the denoised first sensing information.

[0152] As an optional implementation, the first determining module 802 is further configured to perform a short-time Fourier transform on the time-domain information to obtain a first time-frequency spectrum; and / or perform a wavelet transform on the time-domain information to obtain a second time-frequency spectrum.

[0153] As an optional implementation, the second determining module 803 is used to input the first temporal spectrum into the pre-trained first object recognition model to determine the first classification information corresponding to the first temporal spectrum; and / or to input the second temporal spectrum into the pre-trained second object recognition model to determine the second classification information corresponding to the second temporal spectrum.

[0154] As an optional implementation, the third determining module 804 is used to determine the type information of the object to be identified based on the first classification information and / or the second classification information.

[0155] As an optional implementation, the second determining module 803 is further used to train a first object recognition model. Specifically, the second determining module 803 is used to acquire a first sample dataset, which includes a first time-frequency spectrum corresponding to each sample object and historical classification information corresponding to each sample object; construct a first preset machine learning model and determine the first preset machine learning model as the first current machine learning model; predict classification information for the first time-frequency spectrum based on the first current machine learning model to determine the predicted classification information corresponding to the first time-frequency spectrum; determine a loss value based on the predicted classification information and historical classification information corresponding to the first time-frequency spectrum; when the loss value is greater than a preset threshold, backpropagation is performed based on the loss value to update the first current machine learning model to obtain an updated machine learning model, and the updated machine learning model is re-determined as the first current machine learning model; repeat the steps: predict classification information for the first time-frequency spectrum based on the first current machine learning model to determine the predicted object class information corresponding to the first time-frequency spectrum; when the loss value is less than or equal to the preset threshold, the first current machine learning model is determined as the first object recognition model.

[0156] As an optional implementation, the second determining module 803 is further used to train a second object recognition model. Specifically, the second determining module 803 is used to acquire a second sample dataset, which includes a second time-frequency spectrum corresponding to each sample object and historical classification information corresponding to each sample object; construct a second preset machine learning model and determine the second preset machine learning model as the second current machine learning model; predict classification information for the second time-frequency spectrum based on the second current machine learning model to determine the predicted classification information corresponding to the second time-frequency spectrum; determine a loss value based on the predicted classification information and historical classification information corresponding to the second time-frequency spectrum; when the loss value is greater than a preset threshold, backpropagation is performed based on the loss value to update the second current machine learning model to obtain an updated machine learning model, and the updated machine learning model is re-determined as the second current machine learning model; repeat the steps: predict classification information for the second time-frequency spectrum based on the second current machine learning model to determine the predicted object class information corresponding to the second time-frequency spectrum; when the loss value is less than or equal to the preset threshold, the second current machine learning model is determined as the second object recognition model.

[0157] As an optional implementation, the sensor information acquisition module 801 is also used to acquire second sensor information of the object to be identified; the second sensor information is the information of the radar signal after being reflected by the object to be identified.

[0158] As an optional implementation, the device further includes:

[0159] The fourth determining module 805 is used to determine the distance information and instance information between the object to be identified and the current position based on the second sensing information.

[0160] The fusion processing module 806 is used to perform fusion processing on the object to be identified based on distance information, instance information, and category information.

[0161] The apparatus and method embodiments in this invention are based on the same application concept.

[0162] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, servers, or similar computing devices. Taking running on a server as an example, Figure 9 This is a hardware structure block diagram of a server for an object recognition method provided in an embodiment of this application. For example... Figure 9 As shown, the server 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 910 (CPUs 910 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 930 for storing data, and one or more storage media 920 (e.g., one or more mass storage devices) for storing application programs 923 or data 922. The memory 930 and storage media 920 may be temporary or persistent storage. The program stored in the storage media 920 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 910 may be configured to communicate with the storage media 920 and execute the series of instruction operations stored in the storage media 920 on the server 900. Server 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input / output interfaces 940, and / or one or more operating systems 921, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0163] The input / output interface 940 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 900. In one example, the input / output interface 940 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 940 may be a radio frequency (RF) module for wireless communication with the Internet.

[0164] Those skilled in the art will understand that Figure 9 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 900 may also include... Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown.

[0165] Embodiments of this application also provide an object recognition device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the object recognition method.

[0166] The embodiments of this application also provide a storage medium, which can be disposed in a server to store at least one instruction, at least one program, code set or instruction set related to implementing an object recognition method in the method embodiments. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the object recognition method provided in the above method embodiments.

[0167] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0168] As can be seen from the embodiments of the object recognition method, apparatus, device and storage medium provided in this application, the first sensing information of the object to be recognized is obtained. The first sensing information is the information of the terahertz electromagnetic wave after being reflected by the object to be recognized. The time spectrum corresponding to the first sensing information is determined. The time spectrum is input into the pre-trained object recognition model to determine the classification information corresponding to the time spectrum. The type information of the object to be recognized is determined according to the classification information. In this way, the limitation of traditional radar in being unable to recognize and classify objects can be avoided, ensuring the accuracy and efficiency of object recognition and detection, while reducing the dependence on visual sensors.

[0169] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0170] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0171] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0172] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An object recognition method, characterized in that, The method includes: Acquire first sensing information of the object to be identified in the vehicle's driving environment; the first sensing information is information about terahertz electromagnetic waves reflected by the object to be identified; Determine the time-frequency spectrum corresponding to the first sensing information; the time-frequency spectrum is determined based on short-time Fourier transform and / or wavelet transform of the time-domain information corresponding to the first sensing information; The time-spectrum image is input into the trained object recognition model to determine the classification information corresponding to the time-spectrum image; The category information of the object to be identified is determined based on the classification information.

2. The object recognition method according to claim 1, characterized in that, Determining the time-spectrum map corresponding to the first sensing information includes: The first sensing information is denoised to obtain the denoised first sensing information. The first sensor information after noise reduction is converted into time domain information; The time-domain information is used to determine the time-spectrum diagram.

3. The object recognition method according to claim 2, characterized in that, The step of denoising the first sensing information to obtain denoised first sensing information includes: Wavelet decomposition is performed on the first sensing information to obtain the scaling coefficients of the first sensing information; The scaling coefficients are thresholded to obtain the thresholded scaling coefficients. Wavelet reconstruction is performed on the scale coefficients after threshold processing to obtain the first sensing information after denoising.

4. The object recognition method according to claim 2, characterized in that, Determining the time-spectrum map based on the time-domain information includes: Perform a short-time Fourier transform on the time-domain information to obtain a first time-frequency spectrum. and / or; The time-domain information is subjected to wavelet transform to obtain the second time-spectrum diagram.

5. The object recognition method according to claim 4, characterized in that, The step of inputting the time-spectrum image into a pre-trained object recognition model to determine the classification information corresponding to the time-spectrum image includes: The first time-spectrum image is input into the trained first object recognition model to determine the first classification information corresponding to the first time-spectrum image; and / or; The second time-spectrum image is input into the pre-trained second object recognition model to determine the second classification information corresponding to the second time-spectrum image.

6. The object recognition method according to claim 5, characterized in that, Determining the category information of the object to be identified based on the classification information includes: The category information of the object to be identified is determined based on the first classification information and / or the second classification information.

7. The object recognition method according to claim 5, characterized in that, The object recognition method further includes the step of training the first object recognition model: The first object recognition model obtained through training includes: Obtain a first sample dataset, which includes a first time-frequency spectrum corresponding to each sample object among multiple sample objects and historical classification information corresponding to each sample object; Construct a first preset machine learning model, and determine the first preset machine learning model as the first current machine learning model; Based on the first current machine learning model, classification information is predicted for the first time-spectrum image to determine the predicted classification information corresponding to the first time-spectrum image. Based on the predicted classification information and historical classification information corresponding to the first time-spectrum map, the loss value is determined; When the loss value is greater than a preset threshold, backpropagation is performed based on the loss value to update the first current machine learning model to obtain an updated machine learning model, and the updated machine learning model is re-determined as the first current machine learning model; repeat the steps: based on the first current machine learning model, classification information prediction is performed on the first time-spectrum map to determine the predicted object class information corresponding to the first time-spectrum map; When the loss value is less than or equal to the preset threshold, the first current machine learning model is determined as the first object recognition model.

8. The object recognition method according to claim 5 or 7, characterized in that, The object recognition method further includes the step of training the second object recognition model: The second object recognition model obtained through training includes: Obtain a second sample dataset, which includes a second time-frequency spectrum corresponding to each sample object among multiple sample objects and historical classification information corresponding to each sample object; Construct a second preset machine learning model, and determine the second preset machine learning model as the second current machine learning model; Based on the second current machine learning model, classification information is predicted for the second time-spectrum map to determine the predicted classification information corresponding to the second time-spectrum map; Based on the predicted classification information and historical classification information corresponding to the second time-frequency spectrum, the loss value is determined; When the loss value is greater than a preset threshold, backpropagation is performed based on the loss value to update the second current machine learning model to obtain an updated machine learning model. The updated machine learning model is then re-determined as the second current machine learning model. The following steps are repeated: based on the second current machine learning model, classification information is predicted for the second time-spectrum image to determine the predicted object class information corresponding to the second time-spectrum image. When the loss value is less than or equal to the preset threshold, the second current machine learning model is determined as the second object recognition model.

9. The object recognition method according to claim 1, characterized in that, The object recognition method further includes: Acquire second sensing information of the object to be identified; the second sensing information is the information of the radar signal reflected by the object to be identified; Based on the second sensing information, the distance information and instance information between the object to be identified and the current position are determined; The object to be identified is fused based on the distance information, the instance information, and the category information.

10. An object recognition device, characterized in that, The device includes: The sensor information acquisition module is used to acquire first sensor information of the object to be identified in the vehicle's driving environment; the first sensor information is information of terahertz electromagnetic waves reflected by the object to be identified. The first determining module is used to determine the time-frequency spectrum corresponding to the first sensing information; the time-frequency spectrum is determined based on short-time Fourier transform and / or wavelet transform of the time-domain information corresponding to the first sensing information. The second determining module is used to input the time-spectrum image into the trained object recognition model and determine the classification information corresponding to the time-spectrum image. The third determining module is used to determine the type information of the object to be identified based on the classification information.

11. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded by the processor and executed as the object recognition method as described in any one of claims 1-9.

12. A computer storage medium, characterized in that, The computer storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the object recognition method as described in any one of claims 1-9.

Images