A robot control system based on scene recognition

Abstract

The application discloses a robot control system based on scene recognition, comprising: a first acquisition module, configured to acquire a scene image of an area where a robot is located; a first determination module, configured to perform image recognition on the scene image to determine first scene information; a second acquisition module, configured to acquire an audio signal of the area where the robot is located; a second determination module, configured to perform audio recognition on the audio signal to determine second scene information; a fusion module, configured to perform information fusion on the first scene information and the second scene information to determine target scene information of the area where the robot is located; a construction module, configured to construct a 3D scene according to the target scene information; and a third determination module, configured to perform simulation according to the 3D scene and a simulation model constructed in advance, determine a control strategy based on a simulation result, and execute the control strategy. The application improves the accuracy of scene recognition, and further improves the accuracy of a control strategy determined based on scene recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and particularly relates to a robot control system based on scene recognition. BACKGROUND

[0002] At present, with the increasing requirements of robots, the robots cannot only work in a single scene, and need to take different control strategies based on different scenes to work in different scenes. In the prior art, when the robot performs scene recognition, only image signals are used for recognition, which may lead to inaccurate scene recognition and inaccurate control strategies determined. SUMMARY

[0003] The present application aims to at least solve one of the above technical problems. To this end, the present application aims to provide a robot control system based on scene recognition, which improves the accuracy of scene recognition and the accuracy of control strategies determined based on scene recognition.

[0004] To achieve the above-mentioned purpose, the embodiment of the present application provides a robot control system based on scene recognition, comprising:

[0005] A first acquisition module is configured to acquire a scene image of a region where a robot is located;

[0006] A first determination module is configured to perform image recognition on the scene image to determine first scene information;

[0007] A second acquisition module is configured to acquire an audio signal of the region where the robot is located;

[0008] A second determination module is configured to perform audio recognition on the audio signal to determine second scene information;

[0009] A fusion module is configured to perform information fusion on the first scene information and the second scene information to determine target scene information of the region where the robot is located;

[0010] A construction module is configured to construct a 3D scene according to the target scene information;

[0011] A third determination module is configured to perform simulation simulation according to the 3D scene and a pre-constructed simulation model, determine a control strategy based on a simulation simulation result, and execute the control strategy.

[0012] According to some embodiments of the present application, the first determination module comprises:

[0013] A labeling module is configured to label each recognition frame in the scene image based on a pre-trained object recognition model, and determine an object corresponding to each recognition frame;

[0014] a judgment module, configured to judge whether the object is a preset key object, and determine the recognition box corresponding to the object as the preset key object as a key recognition box;

[0015] a segmentation module, configured to segment the scene image into a plurality of sub-scene images according to the size of the scene image and the plurality of recognition boxes, wherein the number of the recognition boxes included in each sub-scene image is within a preset range and at least one key recognition box is included;

[0016] an identification module, configured to perform image identification on the plurality of sub-scene images to determine first scene information.

[0017] According to some embodiments of the present application, the labeling module comprises:

[0018] a labeling sub-module, configured to label each recognition box in the scene image based on a preset trained object recognition model;

[0019] a processing sub-module, configured to input the convolution features into a full connection network in the object recognition model for channel connection processing after performing convolution processing on each recognition box based on a convolution kernel in the object recognition model, to obtain a feature map;

[0020] a matching sub-module, configured to match the feature map with a preset feature map in a preset database, and determine the object corresponding to the preset feature map with the highest matching degree as the object corresponding to the recognition box.

[0021] According to some embodiments of the present application, the judgment module comprises:

[0022] a first determination sub-module, configured to determine the name information of the object;

[0023] a second determination sub-module, configured to perform semantic matching on the name information and the name information of the preset key object, and determine the recognition box corresponding to the object as the key recognition box when it is determined that the matching degree is greater than a preset threshold.

[0024] According to some embodiments of the present application, the identification module comprises:

[0025] a third determination sub-module, configured to determine the association relationship between the recognition box and the key recognition box in each sub-scene image, to form a system set with the key recognition box as the core;

[0026] a conversion sub-module, configured to convert the system set based on a conversion model to obtain a feature value of the system set;

[0027] a fourth determination sub-module, configured to query a preset key recognition box-weight coefficient data table according to the key recognition box included in the sub-scene image to determine a corresponding weight coefficient;

[0028] The fifth determining sub-module is configured to determine the first scene information according to the feature values of the system set in the sub-scene image and the corresponding weight coefficients.

[0029] According to some embodiments of the present application, the training method of the conversion model comprises:

[0030] obtaining a training image sample set, wherein the training image sample set comprises training images corresponding to multiple scenes;

[0031] determining a sample system set and corresponding sample feature values in each training image;

[0032] randomly selecting a training image from the training image sample set and inputting the training image into the conversion model to obtain output feature values;

[0033] performing accuracy inspection on the conversion result by using a loss function according to the sample feature values and the output feature values to obtain a loss function value;

[0034] training parameters of the conversion model by using a back propagation algorithm based on the loss function value until the training is completed.

[0035] According to some embodiments of the present application, the fifth determining sub-module comprises:

[0036] a calculation unit configured to multiply the feature values of the system set in the sub-scene image and the corresponding weight coefficients to obtain preliminary scene information; for a sub-scene image comprising two or more preliminary scene information, the multiple preliminary scene information are fused to obtain fusion results as the sub-scene information corresponding to the sub-scene image;

[0037] a fusion unit configured to perform information fusion again according to the sub-scene information corresponding to several sub-scene images to obtain the first scene information.

[0038] According to some embodiments of the present application, the first determining module further comprises an image preprocessing module configured to perform background non-uniform noise removal processing and illumination normalization processing on the scene image before image recognition.

[0039] According to some embodiments of the present application, the second determining module comprises:

[0040] The extraction module is configured to acquire a plurality of audio scene segments, perform frame processing, extract spectral features of each frame, and form a spectral feature sequence; perform first-order difference processing on the spectral feature sequence to obtain Delta spectral features; perform second-order difference processing on the spectral feature sequence to obtain Double Delta spectral features; take the spectral features, the Delta spectral features, and the Double Delta spectral features as a feature set, perform time-space Gabor filtering processing to obtain a Gabor feature set, construct an audio feature training set, and label a scene label for each audio scene segment;

[0041] The training module is configured to construct an audio scene recognition model, input the Gabor feature set into the audio scene recognition model for training, continuously adjust model parameters of the audio scene recognition model based on a difference between an output result and the labeled scene label until the output result is consistent with the labeled scene label, and indicate that the training is completed.

[0042] The audio recognition module is configured to input an audio signal into the trained audio scene recognition model, and determine second scene information based on an output result.

[0043] According to some embodiments of the present application, the construction module comprises:

[0044] The setting module is configured to set an image model library and an audio model library required by different scenes based on sample JSON files of different scenes.

[0045] The parsing module is configured to convert target scene information into JSON data, and generate a corresponding object; when the type of the object is determined to be an image, a first target field corresponding to the object is determined; and when the type of the object is determined to be audio, a second target field corresponding to the object is determined.

[0046] The fourth determination module is configured to:

[0047] According to the first target field and a preset field of a preset image model in the image model library, a target image model is determined according to a matching result.

[0048] According to the second target field and a preset field of a preset audio model in the audio model library, a target audio model is determined according to a matching result.

[0049] A 3D scene is constructed according to the target image model and the target audio model.

[0050] This invention proposes a robot control system based on scene recognition. It determines first scene information by performing image recognition on the scene image and second scene information by performing audio recognition on the audio signal. The first and second scene information are then fused to determine the target scene information of the robot's location, facilitating the determination of comprehensive scene information. A 3D scene is constructed based on the target scene information, making it easier to determine and reconstruct the robot's environment. Simulation is then performed based on the 3D scene and a pre-built simulation model. Based on the simulation results, an accurate control strategy is determined, improving the accuracy and reliability of robot control.

[0051] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0052] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0053] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0054] Figure 1 This is a block diagram of a scene recognition-based robot control system according to an embodiment of the present invention;

[0055] Figure 2 This is a block diagram of a first determining module according to an embodiment of the present invention;

[0056] Figure 3 This is a block diagram of a labeling module according to an embodiment of the present invention. Detailed Implementation

[0057] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0058] like Figure 1 As shown, this embodiment of the invention proposes a robot control system based on scene recognition, including:

[0059] The first acquisition module is used to acquire scene images of the area where the robot is located;

[0060] The first determining module is used to perform image recognition on the scene image to determine the first scene information;

[0061] The second acquisition module is used to acquire audio signals from the area where the robot is located.

[0062] The second determining module is used to perform audio recognition on the audio signal and determine the second scene information;

[0063] The fusion module is used to fuse the first scene information and the second scene information to determine the target scene information of the area where the robot is located.

[0064] The construction module is used to construct a 3D scene based on the target scene information;

[0065] The third determining module is used to perform simulation based on the 3D scene and the pre-built simulation model, determine the control strategy based on the simulation results, and execute it.

[0066] The working principle of the above technical solution is as follows: In this embodiment, the area is preset, and the corresponding area is determined based on the positioning when determining the robot's location.

[0067] In this embodiment, the target scene information is obtained by information fusion based on the first scene information and the second scene information, including rich scene information such as images and audio.

[0068] In this embodiment, based on the 3D scene and the pre-built simulation model, the simulation robot will produce the following results under control strategy A in the current 3D scene, and the following results under control strategy B. This facilitates accurate determination of the effects of different control strategies on the robot in the current 3D scene, and makes it easier to select the most accurate control strategy.

[0069] The beneficial effects of the above technical solution are as follows: Based on image recognition of the scene image, first scene information is determined; based on audio recognition of the audio signal, second scene information is determined; information fusion of the first scene information and the second scene information is performed to determine the target scene information of the robot's location, which facilitates the determination of comprehensive scene information; a 3D scene is constructed based on the target scene information, which facilitates the determination of the scene in which the robot is located; simulation is performed based on the 3D scene and the pre-built simulation model; and an accurate control strategy is determined based on the simulation results, thereby improving the accuracy and reliability of robot control.

[0070] like Figure 2 As shown, according to some embodiments of the present invention, the first determining module includes:

[0071] The annotation module is used to annotate each bounding box in the scene image based on the pre-trained object recognition model, and to determine the object corresponding to each bounding box;

[0072] A judgment module is configured to judge whether the object is a preset key object, and determine the recognition box corresponding to the object as a key recognition box;

[0073] A segmentation module is configured to segment the scene image into a plurality of sub-scene images according to the size of the scene image and the recognition boxes, wherein the number of the recognition boxes included in each sub-scene image is within a preset range and at least one key recognition box is included;

[0074] An identification module is configured to perform image identification on the plurality of sub-scene images to determine first scene information.

[0075] The working principle of the above technical solution is as follows: in this embodiment, the object recognition model labels each recognition box in the scene image, and each recognition box includes an object, i.e., an object.

[0076] In this embodiment, the key object is a preset object that has a greater impact on the scene.

[0077] In this embodiment, the scene image is segmented into a plurality of sub-scene images, and each sub-scene image includes a number of recognition boxes within a preset range and at least one key recognition box. This facilitates ensuring the accuracy of visual identification and controlling the duration of visual identification of each sub-scene image. Based on the at least one key recognition box, the rationality of sub-scene image division is ensured, there is a clear scene feature, and effective identification is facilitated. The preset number range is 10-12.

[0078] The above technical solution has the following beneficial effects: based on the preset trained object recognition model, each recognition box in the scene image is labeled, and the object corresponding to each recognition box is determined; the scene image is accurately segmented into a plurality of sub-scene images, and based on image identification of the plurality of sub-scene images, accurate identification from a local part is facilitated, the accuracy of determining local features is improved, the efficiency of identification is improved, and the accuracy of the determined first scene information is improved.

[0079] As shown in FIG. 1, according to some embodiments of the present application, the labeling module comprises: Figure 3 A labeling sub-module is configured to label each recognition box in the scene image based on the preset trained object recognition model;

[0080] A processing sub-module is configured to, after performing convolution processing on each recognition box based on the convolution kernel in the object recognition model, input the convolution features into a full connection network in the object recognition model for channel connection processing to obtain a feature map;

[0081]

[0082] ​The matching submodule is configured to match the feature map with preset feature maps in a preset database, and determine an object corresponding to a preset feature map with the highest matching degree as an object corresponding to the recognition box.

[0083] The working principle and beneficial effects of the technical solution are as follows: the labeling submodule is configured to label each recognition box in the scene image based on a preset trained object recognition model; the processing submodule is configured to perform convolution processing on each recognition box based on a convolution kernel in the object recognition model, input convolution features into a full connection network in the object recognition model for channel connection processing, and obtain a feature map; and the matching submodule is configured to match the feature map with preset feature maps in a preset database, and determine an object corresponding to a preset feature map with the highest matching degree as an object corresponding to the recognition box. This facilitates accurate determination of each recognition box on the target image, avoids recognition of irrelevant content, and improves the recognition rate. Matching the feature map with preset feature maps in a preset database facilitates accurate determination of an object corresponding to the recognition box.

[0084] According to some embodiments of the present application, the judging module comprises:

[0085] The first determining submodule is configured to determine name information of the object.

[0086] The second determining submodule is configured to perform semantic matching on the name information and name information of a preset key object, and determine the recognition box corresponding to the object as a key recognition box when it is determined that the matching degree is greater than a preset threshold.

[0087] The working principle and beneficial effects of the technical solution are as follows: the first determining submodule is configured to determine name information of the object; and the second determining submodule is configured to perform semantic matching on the name information and name information of a preset key object, and determine the recognition box corresponding to the object as a key recognition box when it is determined that the matching degree is greater than a preset threshold. Based on semantic recognition matching, it is determined whether the object is a key object, and then the key recognition box is accurately determined.

[0088] According to some embodiments of the present application, the judging module comprises:

[0089] The third determining submodule is configured to determine an association relationship between the recognition box and the key recognition box in each sub-scene image, and form a system set with the key recognition box as the core.

[0090] The conversion submodule is configured to convert the system set based on a conversion model to obtain a feature value of the system set.

[0091] The fourth determining submodule is configured to query a preset key recognition box-weight coefficient data table according to the key recognition box included in the sub-scene image, and determine a corresponding weight coefficient.

[0092] The fifth determination sub-module is configured to determine the first scene information according to the feature values of the system set in the sub-scene image and the corresponding weight coefficients.

[0093] The working principle of the above technical solution is as follows: in this embodiment, the key recognition frame is the key node in the system set, and the other recognition frames are the associated nodes, and the topological relationship graph between the associated nodes and the key node is determined.

[0094] In this embodiment, the feature value corresponding to the system set with the key recognition frame as the core is determined based on the conversion sub-module, and is represented as the image scene feature.

[0095] In this embodiment, different key recognition frames correspond to different weight coefficients, so the weight coefficients of the system set with the key recognition frame as the core are determined accordingly.

[0096] The above technical solution has the following beneficial effects: the system set with the key recognition frame as the core, the determination of the feature values of the system set and the corresponding weight coefficients facilitates the determination of accurate first scene information.

[0097] According to some embodiments of the present application, the training method of the conversion model comprises:

[0098] Obtain a training image sample set, which contains multiple training images corresponding to scenes;

[0099] Determine the sample system set and the corresponding sample feature value in each training image;

[0100] Randomly select a training image from the training image sample set, input it into the conversion model for conversion, and obtain the output feature value;

[0101] According to the sample feature value and the output feature value, the accuracy of the conversion result is verified by a loss function, and a loss function value is obtained;

[0102] Based on the loss function value, the parameters of the conversion model are trained by a back propagation algorithm until the training is completed.

[0103] The working principle and beneficial effects of the technical solution are as follows: a training image sample set is acquired, the training image sample set comprises training images corresponding to multiple scenes; a sample system set in each training image and a corresponding sample feature value are determined; a training image is randomly selected from the training image sample set and is input into a conversion model for conversion to obtain an output feature value; the conversion result is subjected to accuracy inspection by a loss function according to the sample feature value and the output feature value to obtain a loss function value; and the parameters of the conversion model are trained by a back propagation algorithm based on the loss function value until the training is completed. The trained conversion model is obtained, and the accuracy of the feature value obtained by converting the system set is improved.

[0104] According to some embodiments of the application, the fifth determination sub-module comprises:

[0105] The calculation unit is configured to multiply the feature values of the system set in the sub-scene image and the corresponding weight coefficients to obtain preliminary scene information; and for a sub-scene image comprising two or more preliminary scene information, the multiple preliminary scene information is subjected to information fusion, and the fusion result is taken as the sub-scene information corresponding to the sub-scene image.

[0106] The fusion unit is configured to perform information fusion again according to the sub-scene information corresponding to the multiple sub-scene images to obtain the first scene information.

[0107] The beneficial effects of the technical solution are as follows: the sub-scene information corresponding to the sub-scene image is first determined, and the information fusion is performed again according to the sub-scene information corresponding to the multiple sub-scene images to obtain the first scene information, thereby improving the accuracy of the obtained first scene information. For a sub-scene image comprising one preliminary scene information, the preliminary scene information is taken as the sub-scene information of the sub-scene image.

[0108] According to some embodiments of the application, the image preprocessing module is further configured to perform background non-uniform noise removal processing and illumination normalization processing on the scene image before the first determination module performs image recognition on the scene image.

[0109] The beneficial effects of the technical solution are as follows: the noise in the scene image is removed, and the change in brightness is eliminated, thereby improving the accuracy of the obtained scene image.

[0110] According to some embodiments of the application, the second determination module comprises:

[0111] The extraction module is configured to acquire a plurality of audio scene segments, perform frame processing, extract spectral features of each frame, and form a spectral feature sequence; perform first-order difference processing on the spectral feature sequence to obtain Delta spectral features; perform second-order difference processing on the spectral feature sequence to obtain Double Delta spectral features; take the spectral features, the Delta spectral features, and the Double Delta spectral features as a feature set, perform time-space Gabor filtering processing to obtain a Gabor feature set, construct an audio feature training set, and label a scene label for each audio scene segment;

[0112] The training module is configured to construct an audio scene recognition model, input the Gabor feature set into the audio scene recognition model for training, continuously adjust model parameters of the audio scene recognition model based on a difference between an output result and the labeled scene label until the output result is consistent with the labeled scene label, and indicate that the training is completed.

[0113] The audio recognition module is configured to input an audio signal into the trained audio scene recognition model, and determine second scene information based on an output result.

[0114] The working principle of the above technical solution is as follows: in the embodiment, the spectral features, the Delta spectral features, and the Double Delta spectral features are taken as a feature set, and time-space Gabor filtering processing is performed to obtain a Gabor feature set, including: through time-space Gabor filtering, a time modulation filter is represented as a row vector, and is independently convolved with each channel of the feature set; a frequency domain modulation filter is represented as a column vector, and is independently convolved with each frame of the feature set; based on the two convolution results, high-dimensional features are projected onto a low-dimensional space through PCA to obtain the Gabor feature set.

[0115] The above technical solution has the following beneficial effects: the audio features of the audio scene segments are accurately determined, the audio scene recognition model is trained based on the audio features and the labeled scene label, an accurate audio scene recognition model is obtained, an audio signal is input into the trained audio scene recognition model, and second scene information is determined based on an output result, so that accurate second scene information can be obtained.

[0116] According to some embodiments of the present application, the construction module comprises:

[0117] The setting module is configured to set an image model library and an audio model library required by different scenes based on sample JSON files of different scenes.

[0118] The parsing module is used to convert the target scene information into JSON data and generate the corresponding object. When the object type is determined to be an image, the first target field corresponding to the object is determined; when the object type is determined to be audio, the second target field corresponding to the object is determined.

[0119] The fourth determining module is used for:

[0120] The target image model is determined by matching the first target field with the preset fields of the preset image models in the image model library and the matching results are used to determine the target image model.

[0121] The target audio model is determined by matching the second target field with the preset fields of the preset audio models in the audio model library and the matching results.

[0122] Construct a 3D scene based on the target image model and the target audio model.

[0123] The working principle and beneficial effects of the above technical solution are as follows: A setting module is used to set up image model libraries and audio model libraries required for different scenarios based on sample JSON files. A parsing module is used to convert target scene information into JSON data and generate corresponding objects. When the object type is determined to be an image, the first target field corresponding to the object is determined; when the object type is determined to be audio, the second target field corresponding to the object is determined. A fourth determination module is used to match the first target field with preset fields of preset image models in the image model library, and determine the target image model based on the matching result; match the second target field with preset fields of preset audio models in the audio model library, and determine the target audio model based on the matching result; and construct a 3D scene based on the target image model and target audio model. Based on the pre-built image model library and audio model library, and based on the field matching algorithm, the corresponding target image model and target audio model are determined, thereby constructing a 3D scene, improving the construction speed and accuracy.

[0124] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A robot control system based on scene recognition, characterized in that, include: The first acquisition module is used to acquire scene images of the area where the robot is located; The first determining module is used to perform image recognition on the scene image to determine the first scene information; The second acquisition module is used to acquire audio signals from the area where the robot is located. The second determining module is used to perform audio recognition on the audio signal and determine the second scene information; The fusion module is used to fuse the first scene information and the second scene information to determine the target scene information of the area where the robot is located. The construction module is used to construct a 3D scene based on the target scene information; The third determining module is used to perform simulation based on the 3D scene and the pre-built simulation model, determine the control strategy based on the simulation results, and execute it. The first determining module includes: The annotation module is used to annotate each bounding box in the scene image based on the pre-trained object recognition model, and to determine the object corresponding to each bounding box; The judgment module is used to determine whether an object is a preset key object, and to identify the corresponding object as a preset key object as a key recognition box. The segmentation module is used to segment the scene image into several sub-scene images based on the size of the scene image and each recognition box; wherein each sub-scene image includes a number of recognition boxes within a preset range and includes at least one key recognition box; The recognition module is used to perform image recognition on several sub-scene images to determine the first scene information; The identification module includes: The third determination submodule is used to determine the relationship between the recognition box and the key recognition box in each sub-scene image, forming a system set with the key recognition box as the core; The transformation submodule is used to transform the system set based on the transformation model to obtain the feature values ​​of the system set; The fourth determination submodule is used to query a preset key recognition box-weight coefficient data table based on the key recognition boxes included in the sub-scene image to determine the corresponding weight coefficients; The fifth determination submodule is used to determine the first scene information based on the feature values ​​of the system set in the sub-scene image and the corresponding weight coefficients.

2. The robot control system based on scene recognition as described in claim 1, characterized in that, The annotation module includes: The annotation submodule is used to annotate each bounding box in the scene image based on a pre-trained object recognition model. The processing submodule is used to perform convolution processing on each recognition box based on the convolution kernel in the object recognition model, and then input the convolution features into the fully connected network in the object recognition model for channel connection processing to obtain the feature map. The matching submodule is used to match the feature map with preset feature maps in a preset database, and determine the object corresponding to the preset feature map with the highest matching degree as the object corresponding to the recognition box.

3. The robot control system based on scene recognition as described in claim 1, characterized in that, The judgment module includes: The first determination submodule is used to determine the name information of the object; The second determination submodule is used to semantically match the name information with the name information of preset key objects. When the matching degree is greater than the preset threshold, the corresponding object is determined as the recognition box of the preset key object.

4. The robot control system based on scene recognition as described in claim 1, characterized in that, Training methods for transformation models include: Obtain a training image sample set, which contains training images corresponding to multiple scenes; Determine the sample system set and corresponding sample feature values ​​in each training image; Training images are randomly selected from the training image sample set and input into the transformation model for transformation to obtain output feature values; Based on the sample feature values ​​and output feature values, the accuracy of the transformation results is tested using a loss function, and the loss function value is obtained. Based on the loss function value, the parameters of the transformation model are trained using the backpropagation algorithm until training is complete.

5. The robot control system based on scene recognition as described in claim 1, characterized in that, The fifth determining submodule includes: The calculation unit is used to multiply the feature values ​​of the system set in the sub-scene image with the corresponding weight coefficients to obtain preliminary scene information; for a sub-scene image that includes two or more preliminary scene information, the multiple preliminary scene information is fused and the fusion result is used as the sub-scene information corresponding to the sub-scene image. The fusion unit is used to perform information fusion again based on the sub-scene information corresponding to several sub-scene images to obtain the first scene information.

6. The robot control system based on scene recognition as described in claim 1, characterized in that, It also includes an image preprocessing module, which is used to remove background non-uniform noise and normalize illumination in the scene image before the first determining module performs image recognition on the scene image.

7. The robot control system based on scene recognition as described in claim 1, characterized in that, The second determining module includes: An extraction module is used to acquire several audio scene segments, perform frame segmentation, extract the spectral features of each frame, and form a spectral feature sequence; perform first-order difference processing on the spectral feature sequence to obtain Delta spectral features; perform second-order difference processing on the spectral feature sequence to obtain Double Delta spectral features; combine the spectral features, Delta spectral features, and Double Delta spectral features as a feature set, and perform spatiotemporal Gabor filtering to obtain a Gabor feature set, construct an audio feature training set, and label each audio scene segment with a scene label; The training module is used to build an audio scene recognition model. The Gabor feature set is input into the audio scene recognition model for training. Based on the difference between the output result and the labeled scene label, the model parameters of the audio scene recognition model are continuously adjusted until the output result is consistent with the labeled scene label, indicating that the training is complete. The audio recognition module is used to input audio signals into a trained audio scene recognition model and determine the second scene information based on the output results.

8. The robot control system based on scene recognition as described in claim 1, characterized in that, Build modules, including: The configuration module is used to configure the image model library and audio model library required for different scenarios based on sample JSON files. The parsing module is used to convert the target scene information into JSON data and generate the corresponding object. When the object type is determined to be an image, the first target field corresponding to the object is determined; when the object type is determined to be audio, the second target field corresponding to the object is determined. The fourth determining module is used for: The target image model is determined by matching the first target field with the preset fields of the preset image models in the image model library and the matching results are used to determine the target image model. The target audio model is determined by matching the second target field with the preset fields of the preset audio models in the audio model library and the matching results. Construct a 3D scene based on the target image model and the target audio model.

Images