Computer vision-based virtual reality scene three-dimensional target detection method and device

Abstract

The application discloses a kind of based on computer vision virtual reality scene three-dimensional target detection method and device, it is related to virtual reality technical field, the method includes that the image in virtual reality scene is carried out gray scale processing, and using virtual reality technology, target in the image after positioning processing;Based on twin network, obtain the virtual reality image obtained after the target in image is framed;Based on neural network, the feature extraction of target in frame is carried out, to realize the classification of target in frame;The extracted feature is input to classifier, and the classification result of target in frame is detected based on classifier.The application can effectively avoid the situation that neural network parameter optimization is easy to fall into local optimum, and simultaneously, gradient vanishing and gradient explosion phenomenon in traditional gradient descent algorithm can also be avoided.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, and specifically to a method and apparatus for detecting three-dimensional targets in virtual reality scenes based on computer vision. Background Technology

[0002] Advances in computer technology have also brought about more modern design assistance methods. Among them, virtual reality technology has brand-new visualization effects and unique interaction methods. The main principle of virtual reality technology is to use computer simulation technology to create a virtual world in which users can immerse themselves and have an immersive experience.

[0003] Augmented reality (AR) technology, which emerged with the development of virtual reality (VR), overlays computer-generated virtual information onto the real world, thereby enhancing the real world. The advantage of AR is that the space experienced by the user is not completely detached from reality, but rather built upon it. Therefore, it can achieve functions such as information enhancement and navigation, and is currently experiencing rapid development and application in industries such as education and training, industrial inspection, and gaming. VR applications must understand their environment, and a key technology in this process is the detection and pose estimation of three-dimensional objects in reality. Three-dimensional object detection refers to detecting specific objects in an image and providing their three-dimensional spatial information.

[0004] Currently, 3D object detection for virtual reality is still in its early stages of development, with most virtual reality systems only detecting simple planes and specific 2D images. Although some technologies exist capable of 3D object recognition and detection, the detection accuracy needs further improvement. Furthermore, existing deep learning-based detection methods are prone to getting stuck in local optima during parameter optimization in neural network models, and are susceptible to gradient vanishing and gradient exploding phenomena. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a method and device for three-dimensional target detection in virtual reality scenes based on computer vision, which can effectively avoid the situation that neural network parameters are prone to getting stuck in local optima. At the same time, it can also avoid the gradient vanishing and gradient explosion phenomena in the traditional gradient descent algorithm.

[0006] To achieve the above objectives, this invention provides a method for detecting 3D targets in virtual reality scenes based on computer vision, specifically including the following steps:

[0007] The images in the virtual reality scene are converted to grayscale, and the target in the processed image is located using virtual reality technology.

[0008] Virtual reality images obtained by mapping targets in images are acquired based on Siamese networks.

[0009] Features are extracted from targets within the bounding box based on neural networks to classify the targets within the bounding box;

[0010] The extracted features are input into a classifier, and the target within the bounding box is detected based on the classification result of the classifier.

[0011] Based on the above technical solution, the specific steps for grayscale processing of images in a virtual reality scene include:

[0012] Based on the correspondence between grayscale models and color image models, images in virtual reality scenes are converted to grayscale.

[0013] The mean filtering method is used to suppress noise in the grayscale image and divide the image into multiple templates of the same size.

[0014] Based on the above technical solution, the grayscale conversion of images in a virtual reality scene based on the correspondence between grayscale models and color image models is specifically as follows:

[0015] The grayscale model G(r,r,r) is mapped to the color image model RGB(R,G,B), and the range of grayscale values ​​is divided.

[0016] According to the weighted average method, the weights of the three primary colors are obtained to calculate the weighted average of RGB, specifically:

[0017] R = G = B = (Q) r R+Q g G+Q b B)

[0018] Among them, Q r Q represents the proportion of R. g Q represents the proportion of G. b Indicates the proportion of B;

[0019] The grayscale value (Gray) is obtained by weighted average calculation, as follows:

[0020] Gray = 0.3R + 0.59G + 0.11B.

[0021] Based on the above technical solution, the noise suppression process for the grayscale image using the mean filtering method is as follows:

[0022]

[0023] Where k(a,b) represents the filtered image obtained after processing with the mean filtering method, ES represents the set of all pixels in the template, N represents the total number of pixels in the set, j(x,y) represents the image after grayscale processing, and x and y represent the resolution of the image after grayscale processing.

[0024] Based on the above technical solutions,

[0025] The backbone network of the twin network adopts the first three layers of VGG-Net;

[0026] When using a Siamese network to process images, a filter is trained corresponding to the side output of each layer. The filter is then used to calculate the confidence map. The three confidence maps are then combined using a linear weighted fusion method to obtain the image, which is then used to mark the bounding boxes of the targets in the image.

[0027] Based on the above technical solutions,

[0028] The filter is a discriminative filter that outputs target feature blocks at each layer of the Siamese network. The filter is trained, and the ideal output y∈R is obtained. M×N It is a Gaussian function that protrudes from the center position, R represents the space of the data, M×N×H represents a three-dimensional space of size M×N×H, and M×N represents a two-dimensional space of size M×N;

[0029] The filter is obtained by minimizing the output ridge regression:

[0030]

[0031] Among them, w l Let represent the l-th channel of filter w, * denote circular convolution, λ represent the regularization parameter, ε represent the filter output, and D represent the number of data channels. This represents the output target feature block of the l-th channel.

[0032] Based on the above technical solutions,

[0033] The image after grayscale processing is searched, and features are extracted using a Siamese network. The extracted features are: The location of the maximum value in the confidence map g obtained through filtering is used as the target location predicted by the classifier.

[0034]

[0035] in, Indicates w 1 Discrete Fourier Transform, F represents the output target feature block of the l-th channel. -1 This represents the function raised to the power of -1.

[0036] The ground truth image of the image after grayscale processing is compared with the image output by the Siamese network, and then updated accordingly. Specifically:

[0037] φ(Z t )=a×φ(Z1)+(1-a)×φ(Z t-1 )

[0038] Wherein, φ(Z) t ) represents the ground truth image to be updated next, a represents the template update learning rate, and φ(Z1) represents the updated ground truth image. t-1 The input truth image.

[0039] Based on the above technical solutions,

[0040] The neural network consists of three layers. The first layer is the input layer, and the number of neurons in the input layer is the same as the pixel value dimension of the input sample. The number of neurons in the second and third layers is the same as that in the first layer.

[0041] The parameter w of the neuron in the neural network ne and parameter b ne Optimization is achieved through a search operator algorithm, with parameter w ne The parameter b represents the weight of the neuron. ne The threshold representing a neuron;

[0042] The optimization steps specifically include:

[0043] Define search operators and set search conditions, where the individual states of the search operators are represented as follows:

[0044] X = [x1, x2, ..., x n ]

[0045] Where X represents the set of individual states of the search operator, x n This represents the individual state of the nth search operator, where n represents the total number of search operators in the search operator population.

[0046] At some moment, the i-th search operator x i Search for x within the visual field. j When the location is x j The positional state is better than x i Position, then to x j Take one step forward in the direction of position to reach x next If the location is not specified, then continue searching for other locations within the field of view. Specifically:

[0047] x j =x i +Visual·Rand()

[0048]

[0049] Where, x i Let x represent the individual state of the i-th search operator. j Let |x| represent the individual state of the j-th search operator, |Visual| represent the search radius of the search operator, Rand() represent a random number between 0 and 1, and ||x| represent the search radius of the search operator. j -x i || represents the distance between the j-th search operator and the i-th search operator, and Step represents the step size of the search.

[0050] Based on the above technical solutions,

[0051] In the search process, each search operator will select the optimal method from the search behavior, clustering behavior, tail chasing behavior and random behavior to ensure that the population of search operators can reach the position closest to the optimal solution.

[0052] The search behavior specifically refers to:

[0053] Suppose the individual state of the i-th search operator at a certain moment is x. i Randomly select a search operator j within its search range, satisfying x j =x i +Visual·Rand(), based on x i The density of preferred solutions Y i and x j The density of preferred solutions Y j :

[0054] If Y i <Y j Then the i-th search operator moves one step in the direction of the j-th search operator, that is...

[0055]

[0056] If Y i ≥Y j If so, then select a new search operator within the search range and determine whether the move condition is met;

[0057] The clustering behavior specifically refers to:

[0058] Suppose the individual state of the i-th search operator at a certain moment is x. i Given the current state, the number of other search operators found is n, and the search operator at the center position is x. c ,like This indicates that the solution density at the center position is high and not crowded, so the i-th search operator moves one step towards the center position; otherwise, the search continues. Here, Y... c Represents the search operator x c The density of preferred solutions, where δ represents the crowding factor;

[0059] The rear-end collision specifically refers to:

[0060] Suppose the individual state of the i-th search operator at a certain moment is x. i In the current state, search for other nearby search operators, and find the search operator j with the highest priority solution density. This indicates that the density of the preferred solutions at search operator j is high and not crowded, and the i-th search operator moves one step in the direction of search operator j; otherwise, the search continues.

[0061] The random behavior specifically refers to:

[0062] The i-th search operator randomly selects a position within its field of view and moves toward it. The position of the next state is:

[0063] x next =x i +Step·Rand().

[0064] The present invention provides a computer vision-based 3D target detection device for virtual reality scenes, comprising:

[0065] The processing module is used to perform grayscale processing on images in virtual reality scenes and to locate targets in the processed images using virtual reality technology.

[0066] The acquisition module is used to acquire virtual reality images obtained by mapping targets in an image based on a Siamese network.

[0067] The classification module is used to extract features from targets within the bounding box based on a neural network, so as to classify the targets within the bounding box.

[0068] The detection module is used to input the extracted features into the classifier and detect the target within the bounding box based on the classification result of the classifier.

[0069] Compared with the prior art, the advantages of the present invention are as follows: by preprocessing the images in the virtual reality scene, then using a Siamese network to obtain the virtual reality image after the target is bounded, then classifying the images within the bounding box, and finally using a classifier for detection and classification, the present invention can effectively avoid the situation that the neural network parameters are prone to getting stuck in local optima. At the same time, the present invention can also avoid the gradient vanishing and gradient explosion phenomena in the traditional gradient descent algorithm. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart of a computer vision-based 3D target detection method for virtual reality scenes according to an embodiment of the present invention;

[0072] Figure 2 This is a framework diagram of a deformation-resistant correlation filter tracking algorithm based on Siamese networks.

[0073] Figure 3 A schematic diagram illustrating feature extraction of targets within a bounding box using a neural network. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0075] This invention provides a method for detecting 3D targets in virtual reality scenes based on computer vision. By preprocessing the images in the virtual reality scene, then using a Siamese network to obtain virtual reality images with bounding boxes for the targets, classifying the images within the bounding boxes, and finally using a classifier for detection and classification, this method can effectively avoid the situation where neural network parameters easily get stuck in local optima. At the same time, this invention can also avoid the gradient vanishing and gradient exploding phenomena in traditional gradient descent algorithms.

[0076] See Figure 1 As shown in the figure, the present invention provides a method for detecting 3D targets in a virtual reality scene based on computer vision, which specifically includes the following steps:

[0077] S1: Perform grayscale processing on images in virtual reality scenes and use virtual reality technology to locate targets in the processed images; by performing grayscale processing on images in virtual reality scenes, i.e., preprocessing, the processing process is optimized and the target detection accuracy is improved. After grayscale processing, the color changes of the image will be removed, and only the gray intensity change trend will be retained.

[0078] In this invention, the image in a virtual reality scene is converted to grayscale, and the specific steps include:

[0079] S101: Based on the correspondence between grayscale models and color image models, perform grayscale processing on images in virtual reality scenes;

[0080] S102: The mean filtering method is used to suppress noise in the grayscale image and divide the image into multiple templates of the same size.

[0081] In this invention, based on the correspondence between grayscale models and color image models, images in virtual reality scenes are converted to grayscale, specifically as follows:

[0082] S1011: The grayscale model G(r,r,r) is mapped to the color image model RGB(R,G,B), and the grayscale value range is divided between 0 and 255, where 0 represents a completely black image and 255 represents a completely white image. RGB is a color standard, where R represents red, G represents green, and B represents blue. G(r,r,r) is a grayscale representation standard.

[0083] S1012: Based on the weighted average method, the weights of the three primary colors are obtained to calculate the weighted average of RGB, specifically:

[0084] R = G = B = (Q) r R+Q g G+Q b B)

[0085] Among them, Q r Q represents the proportion of R. g Q represents the proportion of G. b Indicates the proportion of B;

[0086] S1013: Gray level is obtained by weighted average calculation, the calculation method is as follows:

[0087] Gray = 0.3R + 0.59G + 0.11B.

[0088] Based on human visual habits and the actual needs of application scenarios, a large amount of experimental data shows that the ratio of the three primary colors R:G:B = 0.3:0.59:0.1 best matches the subjective perception of the human eye. Therefore, the grayscale value Gray is obtained according to the above weighted average calculation method.

[0089] After grayscale processing of the image, mean filtering is used to suppress noise and improve image quality. The acquired image is divided into several templates of the same size. In this invention, mean filtering is used to suppress noise in the grayscale processed image. The specific calculation process is as follows:

[0090]

[0091] Where k(a,b) represents the filtered image obtained after processing with the mean filtering method, ES represents the set of all pixels in the template, N represents the total number of pixels in the set, j(x,y) represents the image after grayscale processing, and x and y represent the resolution of the image after grayscale processing. Its representation can also be x*y.

[0092] Since the output of the center pixel of the mean template is mainly determined by the low-pass filtering method, and the details of the image target are mostly distributed in the high-frequency region of the image, the choice of template determines the noise suppression capability and the clarity of its details. Therefore, the selection of template size needs to balance noise reduction capability and image detail preservation. After image preprocessing, the image target is located using virtual reality technology based on the processing results.

[0093] S2: A virtual reality image obtained by mapping targets in an image based on a Siamese network;

[0094] This invention provides a deformation-resistant correlation filtering tracking algorithm based on Siamese networks, the algorithm framework of which is as follows: Figure 2 As shown, the preprocessed virtual reality ground truth image and the preprocessed virtual reality image are used as inputs to the Siamese network for further feature extraction. The preprocessed virtual reality ground truth image is a manually labeled virtual reality template image.

[0095] Specifically, the backbone of the Siamese network uses the first three layers of VGG-Net (a deep convolutional neural network). When using the Siamese network to process images, a filter is trained corresponding to the side output of each layer. Filtering calculations are performed to obtain a confidence map. The three confidence maps are then combined using a linear weighted fusion method to obtain an image, which is then used for bounding box processing of the targets in the image.

[0096] Since each layer's output contains a description of the target information, the features output by higher-level network structures contain rich semantic information about the target, enabling the identification of the target's location. The features output by lower-level network structures have higher resolution, allowing for precise target localization. Therefore, a filter is trained on the side output of each layer, and correlation filtering is performed. The three confidence maps are then fused using a linear weighted fusion method to obtain an image, thereby locating the target. This training process is supervised.

[0097] In this invention, the filter is a discriminative filter, which outputs the target feature block at each layer of the Siamese network. The filter is trained, and the ideal output y∈R is obtained. M×NIt is a Gaussian function that protrudes from the center position, R represents the space of the data, M×N×H represents a three-dimensional space of size M×N×H, and M×N represents a two-dimensional space of size M×N;

[0098] In this invention, the filter is obtained by minimizing the output ridge regression:

[0099]

[0100] Among them, w l Let represent the l-th channel of filter w, * denote circular convolution, λ represent the regularization parameter, ε represent the filter output, and D represent the number of data channels. Let represent the output target feature block of the l-th channel. The solution to the above equation is:

[0101]

[0102] in, Indicates w l Discrete Fourier Transform, The discrete Fourier transform representing the conjugate value of the complex value y. express The discrete Fourier transform.

[0103] The image after grayscale processing is searched, and features are extracted using a Siamese network. The extracted features are: The location of the maximum value in the confidence map g obtained through filtering is used as the target location predicted by the classifier.

[0104]

[0105] in, Indicates w 1 Discrete Fourier Transform, F represents the output target feature block of the l-th channel. -1 This represents the function raised to the power of -1.

[0106] Next, the confidence graphs are fused. The fusion method of this invention is a linear weighted fusion.

[0107] Next, the ground truth image is updated. Existing Siamese network algorithms employ an online, non-updated feature template strategy, consistently using the pre-processed virtual reality ground truth image for detection and comparison, resulting in fast detection speed. However, in reality, most detected targets are non-rigid targets that undergo significant deformation. During the detection process, the target's shape continuously changes, and if only the features of the pre-processed virtual reality ground truth image are used for comparison and detection, it can easily cause target detection and localization drift. To improve the robustness of the algorithm, this invention proposes a model update method using the following formula: comparing the pre-processed virtual reality ground truth image with the image currently output by the network and updating accordingly, i.e., generating a new virtual reality ground truth image to replace the previously input pre-processed virtual reality ground truth image.

[0108] That is, the ground truth image of the image after grayscale processing is compared with the image output by the Siamese network, and then updated. Specifically:

[0109] φ(Z t )=a×φ(Z1)+(1-a)×φ(Z t-1 )

[0110] Wherein, φ(Z) t ) represents the ground truth image to be updated next, a represents the template update learning rate, and φ(Z1) represents the updated ground truth image. t-1 The input truth image.

[0111] S3: Features are extracted from the targets within the bounding box using a neural network to classify the targets within the bounding box; that is, ... Figure 3 As shown, a three-layer neural network is first used to further extract features from the target within the bounding box.

[0112] In this invention, the neural network comprises three layers. The first layer is the input layer, and the number of neurons in the input layer is the same as the pixel value dimension of the input sample. The number of neurons in the second and third layers is the same as that in the first layer.

[0113] Unlike traditional neural networks that use backpropagation algorithms for parameter optimization, in this invention, the parameters w of neurons in the neural network... ne and parameter b ne Optimization is achieved through a search operator algorithm, with parameter w ne The parameter b represents the weight of the neuron. ne This represents the threshold of a neuron.

[0114] The optimization steps specifically include:

[0115] S301: Define the search operator and set the search conditions, where the individual state of the search operator is represented as follows:

[0116] X = [x1, x2, ..., x n ]

[0117] Where X represents the set of individual states of the search operator, x n This represents the individual state of the nth search operator, where n represents the total number of search operators in the search operator population.

[0118] S302: At some moment, the i-th search operator x i Search for x within the visual field. j When the location is x j The positional state is better than x i Position, then to x j Take one step forward in the direction of position to reach x next If the location is not specified, then continue searching for other locations within the field of view. Specifically:

[0119] x j =x i +Visual·Rand()

[0120]

[0121] Where, x i Let x represent the individual state of the i-th search operator. j Let |x| represent the individual state of the j-th search operator, |Visual| represent the search radius of the search operator, Rand() represent a random number between 0 and 1, and ||x| represent the search radius of the search operator. j -x i || represents the distance between the j-th search operator and the i-th search operator, and Step represents the step size of the search.

[0122] In this invention, during the search process, each search operator selects the optimal method from the search behavior, clustering behavior, tailing behavior and random behavior to ensure that the population of search operators can reach the position closest to the optimal solution.

[0123] Specifically, the search behavior is as follows:

[0124] Suppose the individual state of the i-th search operator at a certain moment is x. i Randomly select a search operator j within its search range, satisfying x j =x i +Visual·Rand(), based on x i The density of preferred solutions Y i and x j The density of preferred solutions Y j :

[0125] If Y i <Yj Then the i-th search operator moves one step in the direction of the j-th search operator, that is...

[0126]

[0127] If Y i ≥Y j If so, a new search operator is selected within the search range, and it is determined whether the move condition is met.

[0128] Specifically, group behavior includes:

[0129] Suppose the individual state of the i-th search operator at a certain moment is x. i Given the current state, the number of other search operators found is n, and the search operator at the center position is x. c ,like This indicates that the solution density at the center position is high and not crowded, so the i-th search operator moves one step towards the center position; otherwise, the search continues. Here, Y... c Represents the search operator x c The density of preferred solutions, where δ represents the crowding factor.

[0130] Specifically, rear-end collisions are defined as follows:

[0131] Suppose the individual state of the i-th search operator at a certain moment is x. i In the current state, search for other nearby search operators, and find the search operator j with the highest priority solution density. This indicates that the density of preferred solutions at search operator j is high and not crowded. The i-th search operator moves one step in the direction of search operator j. Otherwise, the search continues.

[0132] Specifically, random behavior is as follows:

[0133] The i-th search operator randomly selects a position within its field of view and moves toward it. The position of the next state is:

[0134] x next =x i +Step·Rand().

[0135] S4: Input the extracted features into the classifier, and detect the target within the bounding box based on the classifier's classification result. That is, input the sample features extracted in step S3 into the classifier for classification, and detect the category of the 3D target in the virtual reality scene based on the classifier's output. The classifier used in this invention is the logistic regression algorithm.

[0136] For the computer vision-based virtual reality scene 3D target detection method of the present invention, in step S2, a deformation-resistant correlation filtering tracking method based on Siamese network is proposed. Each side output of each layer is used to train a filter and perform correlation filtering calculation. The three confidence maps are fused by linear weighted fusion to obtain an image, and then the target is located. Through this operation step, shallow feature information and deep feature information can be deeply fused, which can help improve the accuracy of 3D target detection.

[0137] Meanwhile, in step S2, the preprocessed virtual reality ground truth image is compared with the image currently output by the network and updated, that is, a new virtual reality ground truth image is generated to replace the previously input preprocessed virtual reality ground truth image. Through this operation step, the bounding box of three-dimensional targets in the virtual reality scene can be improved, which can help to accurately identify three-dimensional targets.

[0138] In step S3, a neural network model optimized based on a search operator algorithm is proposed to optimize the neuron parameters w. ne and parameter b ne This optimization process effectively avoids the situation where neural network parameters easily get trapped in local optima, and also avoids the gradient vanishing and gradient explosion phenomena in traditional gradient descent algorithms.

[0139] In one possible implementation, the present invention also provides a readable storage medium located in a PLC (Programmable Logic Controller) controller. The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the computer vision-based virtual reality scene 3D target detection method described below:

[0140] The images in the virtual reality scene are converted to grayscale, and the target in the processed image is located using virtual reality technology.

[0141] Virtual reality images obtained by mapping targets in images are acquired based on Siamese networks.

[0142] Features are extracted from targets within the bounding box based on neural networks to classify the targets within the bounding box;

[0143] The extracted features are input into a classifier, and the target within the bounding box is detected based on the classification result of the classifier.

[0144] Storage media may be any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0145] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0146] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] This invention provides a computer vision-based 3D target detection device for virtual reality scenes, comprising a processing module, an acquisition module, a classification module, and a detection module.

[0148] The processing module is used to perform grayscale processing on images in virtual reality scenes and to locate targets in the processed images using virtual reality technology; the acquisition module is used to acquire virtual reality images obtained by processing the bounding boxes of targets in the images based on Siamese networks; the classification module is used to extract features from targets within the bounding boxes based on neural networks to classify the targets within the bounding boxes; the detection module is used to input the extracted features into the classifier and detect the targets within the bounding boxes based on the classification results of the classifier.

[0149] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

[0150] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

Claims

1. A method for detecting 3D targets in virtual reality scenes based on computer vision, characterized in that, Specifically, the following steps are included: The images in the virtual reality scene are converted to grayscale, and the target in the processed image is located using virtual reality technology. Virtual reality images obtained by mapping targets in images are acquired based on Siamese networks. Features are extracted from targets within the bounding box based on neural networks to classify the targets within the bounding box; The extracted features are input into a classifier, and the target within the bounding box is detected based on the classification result of the classifier. The specific steps for performing grayscale processing on images in a virtual reality scene include: Based on the correspondence between grayscale models and color image models, images in virtual reality scenes are converted to grayscale. The mean filtering method is used to suppress noise in the grayscale image, and the image is divided into multiple templates of the same size. The backbone network of the twin network adopts the first three layers of VGG-Net; When using a Siamese network to process images, a filter is trained corresponding to the side output of each layer. The filter is then used to calculate the confidence map. The three confidence maps are then combined using a linear weighted fusion method to obtain an image, which is then used to process the bounding boxes of the targets in the image. Specifically, the ground truth image of the image after grayscale processing is compared with the image output by the Siamese network, and then updated. in, This represents the truth image that will be updated next. This indicates that the template updates the learning rate. This represents the updated truth image. The input ground truth image.

2. The method for detecting 3D targets in a virtual reality scene based on computer vision as described in claim 1, characterized in that, The grayscale conversion of images in a virtual reality scene based on the correspondence between grayscale models and color image models is specifically as follows: In the grayscale model , and in color image models Correspondingly, the range of grayscale values ​​is divided; According to the weighted average method, the weights of the three primary colors are obtained to calculate the weighted average of RGB, specifically: in, express The proportion of which express The proportion of which express The proportion of; Gray levels are obtained by weighted average calculation. The calculation method is as follows: 。 3. The method for detecting 3D targets in a virtual reality scene based on computer vision as described in claim 1, characterized in that, The noise suppression process for the grayscale-processed image using mean filtering is as follows: in, This represents the filtered image obtained after processing with the mean filtering method. This represents the set of all pixels in the template. This represents the total number of pixels in the set. This represents the image after grayscale conversion. and This indicates the resolution of the image after grayscale processing.

4. The method for detecting 3D targets in a virtual reality scene based on computer vision as described in claim 1, characterized in that: The filter is a discriminative filter that outputs target feature blocks at each layer of the Siamese network. Training the filter, ideal output It is a Gaussian function that bulges out at the center. The space representing the data, Indicates the size is Three-dimensional space Indicates the size is Two-dimensional space; The filter is obtained by minimizing the output ridge regression: in, Represents filter The aisle, Represents circular convolution. Represents the regularization parameter. Indicates the output of the filter. Indicates the number of data channels. Indicates the first The output target feature block of the channel.

5. The method for detecting 3D targets in a virtual reality scene based on computer vision as described in claim 4, characterized in that: The image after grayscale processing is searched, and features are extracted using a Siamese network. The extracted features are: The confidence map obtained through filtering calculation The position of the maximum value is used as the target position predicted by the classifier: in, express Discrete Fourier Transform, Indicates the first The output target feature block of the channel This represents the function raised to the power of -1.

6. The method for detecting 3D targets in a virtual reality scene based on computer vision as described in claim 1, characterized in that: The neural network consists of three layers. The first layer is the input layer, and the number of neurons in the input layer is the same as the pixel value dimension of the input sample. The number of neurons in the second and third layers is the same as that in the first layer. The parameters of neurons in the neural network and parameters Optimization is achieved through a search operator algorithm, with parameters... The weights of neurons, parameters The threshold representing a neuron; The optimization steps specifically include: Define search operators and set search conditions, where the individual states of the search operators are represented as follows: in, This represents the set of individual states of the search operator. Indicates the first The individual state of a search operator This represents the total number of search operators in the search operator population; At a certain moment, the A search operator In the field of vision Internal search found When the location is, if The positional state is better Location, then towards Take one step forward in the direction of location, and arrive at If the location is not specified, then continue searching for other locations within the field of view. Specifically: in, Indicates the first The individual state of a search operator Indicates the first The individual state of a search operator This represents the search radius of the search operator. Represents a random number between 0 and 1. Indicates the first The search operator and the first Distance between search operators Indicates the step size of the search.

7. The method for detecting 3D targets in a virtual reality scene based on computer vision as described in claim 6, characterized in that: In the search process, each search operator will select the optimal method from the search behavior, clustering behavior, tail chasing behavior and random behavior to ensure that the population of search operators can reach the position closest to the optimal solution. The search behavior specifically refers to: Assume the first The individual state of a search operator at a certain moment is Randomly select a search operator within its search range. ,satisfy Based on Priority solution density and Priority solution density : like Then the first The search operator directs the first Each search operator moves one step forward in its direction, that is... ； like If so, then select a new search operator within the search range and determine whether the move condition is met; The clustering behavior specifically refers to: Assume the first The individual state of a search operator at a certain moment is The number of other search operators found in the current state is The search operator at the center position is ,like This indicates that the central location has a higher density of preferred solutions and is not crowded. Each search operator moves one step towards the center position; otherwise, the search continues. Represents the search operator The density of priority solutions Indicates the crowding factor; The rear-end collision specifically refers to: Assume the first The individual state of a search operator at a certain moment is In the current state, search for other nearby search operators and find the search operator with the highest priority solution density. ,like This indicates the search operator. The priority solution at point 1 has a high density and is not crowded. A search operator to a search operator Move one step in the current direction, or continue searching if the direction is reversed. The random behavior specifically refers to: No. Each search operator randomly selects a position within its field of view and moves toward it. The position of the next state is: 。 8. A computer vision-based 3D target detection device for virtual reality scenes, characterized in that, include: The processing module is used to perform grayscale processing on images in virtual reality scenes and to locate targets in the processed images using virtual reality technology. The acquisition module is used to acquire virtual reality images obtained by mapping targets in an image based on a Siamese network. The classification module is used to extract features from targets within the bounding box based on a neural network, so as to classify the targets within the bounding box. The detection module is used to input the extracted features into the classifier and detect the target within the bounding box based on the classification result of the classifier. The specific steps for performing grayscale processing on images in a virtual reality scene include: Based on the correspondence between grayscale models and color image models, images in virtual reality scenes are converted to grayscale. The mean filtering method is used to suppress noise in the grayscale image, and the image is divided into multiple templates of the same size. The backbone network of the twin network adopts the first three layers of VGG-Net; When using a Siamese network to process images, a filter is trained corresponding to the side output of each layer. The filter is then used to calculate the confidence map. The three confidence maps are then combined using a linear weighted fusion method to obtain an image, which is then used to process the bounding boxes of the targets in the image. Specifically, the ground truth image of the image after grayscale processing is compared with the image output by the Siamese network, and then updated. in, This represents the truth image that will be updated next. This indicates that the template updates the learning rate. This represents the updated truth image. The input ground truth image.

Images