Obstacle detection method, obstacle model acquisition method and device and smart home equipment

Abstract

The invention relates to the field of smart home equipment, and discloses an obstacle detection method, a model acquisition method and device thereof and smart home equipment, and the obstacle detection model acquisition method comprises the steps: constructing a teacher network and a student network based on a target detection network; inputting a sample image collected by the smart home device into a student network for processing to obtain student features output by the student network in different network layers; obtaining text information of the sample image, inputting the sample image and the text information into a teacher network for processing, and obtaining teacher features output by the teacher network in different network layers; and calculating network training loss according to the teacher features and the student features, adjusting network parameters by using the training loss, and taking the trained student network as an obstacle detection model. According to the method, the problem of high training difficulty of the obstacle detection model deployed to the smart home equipment can be solved.

Description

Technical Field

[0001] The present application relates to the field of smart home devices, and in particular to an obstacle detection method and a model acquisition method, device and smart home device. Background Art

[0002] With the development of artificial intelligence technology, smart home devices have gradually become popular, such as sweepers, which have become common tools in daily life as a powerful assistant for home cleaning. By introducing deep learning and computer vision technology, smart home devices can complete more tasks autonomously.

[0003] Take a sweeper as an example. Its obstacle detection function is crucial to its navigation and obstacle avoidance capabilities. However, when designing such deep learning models, there are often bottlenecks in performance optimization. For example, firstly, due to the cost of model deployment and computing resources, sweepers usually use models with a small number of parameters, which may affect the accuracy of obstacle detection in complex environments. Secondly, due to the difference between the camera perspective of a sweeper and the acquisition perspective of traditional data sets, it is difficult to achieve ideal optimization effects by pre-training models based on large-scale public image data sets. Summary of the invention

[0004] In view of this, the present application provides an obstacle detection method and a model acquisition method, device and smart home device thereof. The obstacle detection model trained using the method of the present application can have better detection effect without significantly increasing the computing burden of the device.

[0005] In a first aspect, an embodiment of the present application provides a method for acquiring an obstacle detection model, comprising:

[0006] Build the teacher network and student network based on the target detection network;

[0007] Inputting sample images collected by smart home devices into the student network for processing, and obtaining student features output by the student network at different network layers;

[0008] Acquire text information of the sample image, input the sample image and the text information into the teacher network for processing, and obtain teacher features output by the teacher network at different network layers;

[0009] The network training loss is calculated according to the teacher characteristics and the student characteristics, so as to adjust the network parameters by using the training loss, and the student network after the training is used as an obstacle detection model.

[0010] In some embodiments, the teacher network and the student network both include: an image feature extraction layer, a feature fusion layer, and a prediction output layer connected in sequence;

[0011] The student features include: image fusion features output by the feature fusion layer in the student network and first prediction features output by the prediction output layer;

[0012] The teacher features include: image-text fusion features output by the feature fusion layer in the teacher network and second prediction features output by the prediction output layer.

[0013] In some embodiments, the training loss includes: a prediction loss of each of the teacher network and the student network, and a distillation loss between the teacher network and the student network, wherein the distillation loss includes an intermediate feature alignment loss and a prediction feature alignment loss;

[0014] The calculating the network training loss according to the teacher characteristics and the student characteristics includes:

[0015] According to the first prediction feature and the second prediction feature, respectively calculating the prediction loss of the teacher network and the student network for the sample image, and calculating the prediction feature alignment loss between the teacher network and the student network;

[0016] The intermediate feature alignment loss between the teacher network and the student network is calculated based on the image-text fusion feature and the image fusion feature.

[0017] In some embodiments, the training loss is obtained by weighted summing the prediction loss of the teacher network, the prediction loss of the student network, and the distillation loss.

[0018] In some embodiments, the distillation loss is obtained by weighted summing the intermediate feature alignment loss and the predicted feature alignment loss.

[0019] In some embodiments, the calculating the prediction feature alignment loss between the teacher network and the student network comprises:

[0020] A similarity between the first prediction feature and the second prediction feature is calculated, and the prediction feature alignment loss is calculated based on the similarity.

[0021] In some embodiments, the calculating the intermediate feature alignment loss between the teacher network and the student network comprises:

[0022] A cross entropy loss between the image-text fusion feature and the image fusion feature is calculated as the intermediate feature alignment loss.

[0023] In some embodiments, the inputting of the sample images collected by the smart home device into the student network for processing to obtain the student features output by the student network at different network layers includes:

[0024] Using the image feature extraction layer in the student network to extract features from sample images collected by the smart home device, to obtain a first image feature;

[0025] Using the feature fusion layer to perform feature fusion on the first image features to obtain the image fusion features;

[0026] The image fusion feature is predicted using the prediction output layer to obtain the first prediction feature output by the student network.

[0027] In some embodiments, the teacher network further includes a text feature extraction layer, and the sample image and the text information are input into the teacher network for processing to obtain teacher features output by the teacher network at different network layers, including:

[0028] Using the image feature extraction layer and the text feature extraction layer in the teacher network to extract features of the sample image and the text information respectively, to obtain second image features and text features;

[0029] Using the feature fusion layer to fuse the second image feature and the text feature to obtain the image-text fusion feature;

[0030] The image-text fusion feature is predicted using the prediction output layer to obtain the second prediction feature output by the teacher network.

[0031] In some embodiments, the teacher network and the student network are both constructed using the target detection network model of the YOLO series;

[0032] The network model of the yolo series includes Backbone network, Neck network and Head network, which serve as the image feature extraction layer, the feature fusion layer and the prediction output layer respectively;

[0033] Among them, the network parameter amount in the teacher network is greater than the network parameter amount in the student network.

[0034] In some embodiments, the obtaining text information of the sample image includes:

[0035] The sample image is input into a large language model to generate a corresponding image description text; a text feature extraction model is used to extract features from the image description text to obtain text features; and the text features are used to be input into the feature fusion layer in the teacher network.

[0036] In some embodiments, when the teacher network also includes a text feature extraction layer, the sample image is input into a large language model to generate corresponding image description text; the image description text is used to input into the text feature extraction layer in the teacher network for text feature extraction.

[0037] In some embodiments, the text feature extraction model or the text feature extraction layer is constructed using a pre-trained language model.

[0038] In a second aspect, an embodiment of the present application further provides an obstacle detection method, comprising:

[0039] Obtain the image to be tested collected by the smart home device in real time;

[0040] The image to be tested is input into an obstacle detection model for processing to obtain an obstacle detection result; wherein the obstacle detection model is obtained by using the model acquisition method.

[0041] In a third aspect, an embodiment of the present application further provides an obstacle detection model acquisition device, comprising:

[0042] A network construction module is used to construct a teacher network and a student network based on the target detection network;

[0043] A student network module is used to process sample images collected by smart home devices input into the student network to obtain student features output by the student network at different network layers;

[0044] A text information acquisition module, used to acquire text information of the sample image;

[0045] A teacher network module, used for processing the sample image and the text information input into the teacher network to obtain teacher features output by the teacher network at different network layers;

[0046] The network training module is used to calculate the network training loss according to the teacher characteristics and the student characteristics, so as to adjust the network parameters by using the training loss, and use the student network after the training as an obstacle detection model.

[0047] In a fourth aspect, an embodiment of the present application further provides a smart home device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the obstacle detection model acquisition method or the obstacle detection method.

[0048] In a fifth aspect, an embodiment of the present application further provides a readable storage medium storing a computer program, which, when executed, implements the obstacle detection model acquisition method or obstacle detection method.

[0049] The embodiments of the present application have the following advantages:

[0050] This application proposes a method for obtaining an obstacle detection model, which first constructs a teacher network and a student network based on a target detection network; inputs a sample image into the student network for processing to obtain student features; at the same time, the sample image and the text information of the sample image are input into the teacher network for processing, that is, a multimodal input is formed to obtain teacher features; then, the network training loss is calculated according to the teacher features and the student features, and the network parameters are adjusted using the training loss, and the student network after training is used as an obstacle detection model. This method combines knowledge distillation and multimodal teacher network technology, and uses a lightweight target detection network to build a model. On the one hand, it solves the problem of model deployment in smart home devices well, and on the other hand, it provides a new idea for the training of obstacle detection models. It not only solves the problem of differences in sample images collected from camera perspectives, but also effectively improves the detection performance of the trained obstacle detection model, and enhances its obstacle detection capabilities in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A schematic diagram of the structure of a smart home device according to an embodiment of the present application;

[0053] Figure 2 A flowchart of the obstacle detection model acquisition method of the embodiment of the present application;

[0054] Figure 3 A schematic diagram of the structure of a teacher network and a student network in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of the teacher network and student network built based on yolov6;

[0056] Figure 5 A schematic diagram of the distillation loss of the teacher network and the student network in an embodiment of the present application;

[0057] Figure 6A flow chart of calculating network training loss in an embodiment of the present application;

[0058] Figure 7 A flowchart of an obstacle detection method according to an embodiment of the present application;

[0059] Figure 8 Schematic diagram of the structure of the obstacle detection model acquisition device of the embodiment of the present application.

[0060] Description of main component symbols:

[0061] 10-smart home device; 11-processor; 12-memory; 13-sensing unit; 14-execution unit;

[0062] 100 - obstacle detection model acquisition device; 110 - network construction module; 120 - student network module; 130 - text information acquisition module; 140 - teacher network module; 150 - network training module. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0064] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0065] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present application, are intended only to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing items.

[0066] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0067] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0068] Since the smart home device 10 is limited by the cost of deploying the obstacle detection model and the particularity of the camera viewing angle installed on it, such as the camera on the sweeper has a viewing angle limitation in the height direction, the design and training of the obstacle detection model of the device becomes a problem. To this end, the inventor proposes a solution combining knowledge distillation and multimodal input. Specifically, a knowledge distillation method based on a multimodal input teacher network is proposed to obtain an obstacle detection model, aiming to solve the problem of obstacle detection model training for the smart home device 10.

[0069] The technical solution of the present application will be described below in conjunction with some embodiments.

[0070] Figure 1 A structural schematic diagram of a smart home device 10 according to an embodiment of the present application is shown.

[0071] Exemplarily, the smart home device 10 includes a processor 11, a memory 12, a sensing unit 13 and an execution unit 14, wherein the memory 12 stores a computer program, and the processor 11 runs the computer program, thereby enabling the smart home device 10 to execute the obstacle detection model acquisition method or the obstacle detection method of the embodiment of the present application.

[0072] Among them, the processor 11 can be an integrated circuit chip with signal processing capabilities. The processor 11 can be a general-purpose processor 11, including a central processing unit (CPU), a graphics processing unit (GPU) and a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or at least one of other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0073] The memory 12 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 12 is used to store a computer program, and the processor 11 may execute the computer program accordingly after receiving an execution instruction.

[0074] The sensing unit 13 is mainly used to transmit external environmental information to the control system of the smart home device 10, so as to provide necessary conditions for the smart home device 10 in scenarios such as navigation, map building or recharging. For example, the sensing unit 13 includes, but is not limited to, a camera, etc. Taking a sweeper as an example, the camera can be set in front of the sweeper to capture the environmental information of the front area in real time during the movement of the sweeper, so as to provide the control system of the smart home device 10 with operations such as obstacle recognition, detection, and avoidance.

[0075] The execution unit 14 is mainly used to enable the smart home device 10 to perform corresponding actions or tasks. For example, for a sweeper, its execution unit 14 includes a cleaning mechanism and a walking mechanism. For example, the cleaning mechanism may include but is not limited to a spray cleaning component, a dirt removal component, etc., and the walking mechanism may include but is not limited to side wheels, track wheels, tracks, rollers, etc. It can be understood that the specific structure of the execution unit 14 can be determined according to the actual needs of the smart home device 10, and will not be described in detail here.

[0076] It can be understood that the above-mentioned smart home device 10 may include but is not limited to a sweeping machine (also known as a cleaning robot), a mobile accompanying machine, a food delivery machine, etc.

[0077] Figure 2 A flow chart of the obstacle detection model acquisition method according to an embodiment of the present application is shown.

[0078] Exemplarily, the obstacle detection model acquisition method includes the following steps:

[0079] S110, constructing a teacher network and a student network based on the target detection network.

[0080] The target detection network refers to a convolutional neural network that can be used in target detection scenarios, and in this application, it is mainly used to implement obstacle detection of the smart home device 10. For example, for a sweeper, when walking, it may encounter various obstacles, including but not limited to sofas, tables and chairs, walls, doors, etc.

[0081] Demonstratively, a teacher network and a student network are constructed based on the target detection network, so as to improve the performance of the student network by using knowledge distillation technology. The trained student network is used as the obstacle detection model for deployment. Among them, knowledge distillation is to train a student network with a smaller number of parameters to imitate a teacher network with a larger number of parameters, so that the student network can learn more discrimination and feature representation capabilities under the condition of limited computing resources.

[0082] Since the trained obstacle detection model needs to be deployed in the smart home device 10, and is limited by the cost and computing resources of model deployment, when selecting the target detection network for constructing the teacher network and the student network, a lightweight convolutional neural network is preferably selected, for example, including but not limited to, the yolo series model or a lightweight derivative model based on the yolo series. Taking the yolo series as an example, different versions can be selected, such as the yolov4 model, the yolov5 model or the yolov6 model, etc., depending on the performance or requirements of the actual scene, which is not limited here.

[0083] In one embodiment, the structures of the convolutional neural networks of the constructed teacher network and the student network may be the same, for example, Figure 3 As shown, the teacher network and the student network both include: an image feature extraction layer, a feature fusion layer, and a prediction output layer connected in sequence. Among them, the image feature extraction layer is used to extract image features, the feature fusion layer is used to fuse multi-dimensional features, and the prediction output layer is used as the output layer of the network to output the prediction results. It should be noted that the network parameters in the teacher network will be greater than the network parameters in the student network. It can be understood that the student network is obtained after network parameter trimming, so it requires fewer computing resources.

[0084] For example, the yolov6 model can be selected to construct the teacher network and the student network respectively. Figure 4 As shown, for the yolov6 series model, it includes a Backbone network, a Neck network and a Head network, which respectively serve as the image feature extraction layer (corresponding to the Backbone network), the feature fusion layer (corresponding to the Neck network) and the prediction output layer (corresponding to the Head network) in the above-mentioned teacher network and student network.

[0085] It can be understood that for the yolov6 model, according to the number of parameters and the number of floating-point operations, it is divided into models with different parameter amounts, for example, there are yolov6-N, yolov6-M, yolov6-L, yolov6-N6, yolov6-L6, etc., among which the model parameter amounts of yolov6-N, yolov6-M, and yolov6-L increase successively, and accordingly, the feature extraction ability of the model is also stronger; yolov6-L6 is a model with stronger feature extraction ability obtained on the basis of yolov6-L. For example, for the teacher network, the yolov6-L6 model can be used to construct it, and the student network can adopt a model with a smaller parameter amount than the yolov6-L6 model, such as the yolov6-N model, etc., which is not limited here.

[0086] It can be understood that for the teacher network and the student network, each network layer will input corresponding feature information. In order to pass the knowledge of the teacher network to the student network, this application will align the corresponding features output by the student network and the teacher network at different network layers to ensure that the student network can better learn the feature representation of the teacher network.

[0087] S120, inputting the sample images collected by the smart home device 10 into the student network for processing, and obtaining the student features output by the student network at different network layers.

[0088] The sample images are used to synchronously train the teacher network and the student network constructed above, and can be acquired by the smart home device 10 through a camera set by itself. Usually, these sample images can be annotated with obstacle information manually or automatically, so as to subsequently calculate the prediction loss information of the teacher network and the student network when outputting the prediction results.

[0089] Exemplarily, the sample images for training are input into the student network so as to be processed in sequence by each network layer in the student network to obtain the corresponding features output by each network layer, which are recorded as student features here to distinguish them from the corresponding features output by the teacher network.

[0090] It should be understood that the features used for subsequent calculation of training loss can be selected from the corresponding features output by each network layer in the student network according to actual needs. For example, in one embodiment, the above-mentioned student features include, but are not limited to, image fusion features output by the feature fusion layer in the student network, and the first prediction features output by the prediction output layer.

[0091] S130, obtaining text information of the sample image, inputting the sample image and the text information into the teacher network for processing, and obtaining teacher features output by the teacher network at different network layers.

[0092] The text information is used to describe the scene content in the image in text form. For example, for an image of a cat next to a table leg, the text information can be described as "a cat is standing next to the table leg, and it is looking at me", etc., thereby extracting more environmental information.

[0093] It should be understood that there are many ways to obtain and describe the above text information. For example, it can be an image description text corresponding to the sample image generated based on a pre-trained large language model (LLM), or it can be a text feature corresponding to the sample image obtained by further extracting text features based on the image description text. The LLM model can be, but is not limited to, Chat-GPT, Wenxinyiyan and other models.

[0094] It is understandable that if the image description text is input into the teacher network as text information, such as Figure 5 As shown, the teacher network should also include a text feature extraction layer for text feature extraction. On the contrary, if the text features are input into the teacher network as text information, the text feature extraction can be performed by a pre-trained text feature extraction model set outside the teacher network.

[0095] For example, in one embodiment, the text feature extraction model or text feature extraction layer can be constructed using an open source pre-trained language model, such as but not limited to a BERT (Bidirectional Encoder Representations from Transformers) model, a GPT (Generative Pre-trained Transformer) model, an ALBERT model derived from a BERT model, etc., which is not limited here.

[0096] Exemplarily, the sample image input into the student network is also input into the teacher network for processing. The difference is that the teacher network also inputs the text information of the sample image to form a multi-model input of the teacher network. It can be understood that by effectively fusing the image and text information, thereby promoting multi-modal information fusion, the accuracy of obstacle detection can be enhanced.

[0097] Based on multimodal input, in one embodiment, the teacher features include, but are not limited to, image-text fusion features output by the feature fusion layer in the teacher network, and second prediction features output by the prediction output layer. It can be understood that the "first" and "second" in the first prediction feature and the second prediction feature are only used to distinguish the prediction features output by the student network from the prediction features output by the teacher network.

[0098] It is worth noting that the operations in the above steps S200 and S300 are not limited to a strict execution order, and can be executed synchronously or sequentially in a preset order. For example, text information can be extracted from a sample image first, and then the sample image and text information can be input into the corresponding network.

[0099] S140, calculating the network training loss according to the teacher characteristics and the student characteristics, so as to adjust the network parameters by using the training loss, and using the student network after the training as the obstacle detection model.

[0100] Among them, network model training is to learn by continuously reducing the value of the loss function. In this application, in order to train the teacher network and the student network, the corresponding teacher features and student features output by each network are used in combination with the constructed loss function to calculate the corresponding training loss, and then each training loss is used to perform gradient back transmission to continuously update the parameters of the network, thereby achieving the purpose of network training.

[0101] In one embodiment, the above-mentioned training loss includes two categories, namely: the prediction loss of the teacher network and the student network for the sample image respectively, and the distillation loss between the teacher network and the student network; wherein the distillation loss is used to evaluate the degree of learning between the teacher network and the student network, and the smaller the distillation loss is, the closer the feature representation ability acquired by the student network is to the teacher network.

[0102] As an optional solution, for the distillation loss, the intermediate feature alignment loss and the predicted feature alignment loss between the teacher network and the student network can be included, such as Figure 5 As shown in Figure 2, it can be understood that this scheme enables the student network to learn the multimodal information in the teacher network more comprehensively by aligning features in multiple dimensions.

[0103] For the above two types of losses, in one implementation, Figure 6 As shown, including:

[0104] S210, respectively calculating the prediction losses of the student network and the teacher network for the sample image according to the first prediction feature and the second prediction feature.

[0105] That is, the prediction loss of the student network is calculated using the first prediction feature and the true label of the sample image; and the prediction loss of the teacher network is calculated using the second prediction feature and the true label of the sample image. The true label refers to the obstacle annotation information in the sample image. Regarding the prediction loss, for example, the cross entropy loss function can be used to calculate, which is not limited here.

[0106] S220, calculating the prediction feature alignment loss between the teacher network and the student network according to the first prediction feature and the second prediction feature.

[0107] The prediction feature alignment loss is used to describe the alignment deviation between the prediction results finally outputted by the teacher network and the student network. For example, the prediction feature alignment loss can be calculated by calculating the similarity between the first prediction feature and the second prediction feature, and based on the similarity.

[0108] For example, in one implementation, cosine similarity may be used for calculation, and thus, the calculation method of the prediction feature alignment loss is described by the expression:

[0109]

[0110] L(Feature)=1-Sim(F S ,F T );

[0111] In the formula, Sim(F S ,F T ) represents the first prediction feature F S and the second predictive feature F T The cosine similarity between them, L(Feature) represents the prediction feature alignment loss.

[0112] S230, calculating the intermediate feature alignment loss between the teacher network and the student network according to the image-text fusion feature and the image fusion feature.

[0113] Among them, the intermediate feature alignment loss is used to describe the alignment deviation between the features output by the intermediate feature fusion layer between the teacher network and the student network. The image-text fusion feature is obtained by fusing the image features and text features extracted from the sample image by the teacher network, while the image fusion feature is obtained by fusing only the image features extracted from the sample image by the student network.

[0114] For example, the cross entropy loss between the image-text fusion feature and the image fusion feature can be calculated as the intermediate feature alignment loss. If described by an expression, we have:

[0115] L(Result)=E(R S ,R T );

[0116] In the formula, L(Result) represents the image fusion feature R S And image text fusion feature R T The cross entropy loss between ; E represents the cross entropy loss function.

[0117] It can be understood that there is no sole limitation on the selection of the above-mentioned various loss functions.

[0118] It should be understood that the above-calculated prediction feature alignment loss and intermediate feature alignment loss are both distillation losses. For example, in one embodiment, the distillation loss can be obtained by weighted summing the above-mentioned intermediate feature alignment loss and prediction feature alignment loss. If described by an expression, that is:

[0119] L(Distillation)=L(Feature)+λ·L(Result);

[0120] In the formula, L (Distillation) represents the distillation loss, and λ is the multi-dimensional feature alignment balance factor, which is used to adjust the weight between the intermediate feature alignment loss and the predicted feature alignment loss. It can be understood that if λ = 1, it means that the weights of the intermediate feature alignment loss and the predicted feature alignment loss are equal.

[0121] Furthermore, the total training loss of the entire teacher network and the student network can be calculated by the prediction loss of the student network and the teacher network respectively, and the distillation loss calculated above.

[0122] For example, in one embodiment, the total training loss can be obtained by weighted summing the prediction loss of the teacher network, the prediction loss of the student network, and the distillation loss. If described by an expression, we have:

[0123] L(Total)=L(Student)+α·L(Teacher)+β·L(Distillation);

[0124] In the formula, L(Total) represents the total training loss; L(Student) is the prediction loss of the student network, and L(Teacher) is the prediction loss of the teacher network; α and β are the loss function balance factors, which are used to adjust the weights of the prediction loss of the student network, the prediction loss of the teacher network, and the distillation loss on the regression of the parameters of the entire network model. It can be understood that if α and β are both 1, it means that the weights of these three losses are equal.

[0125] Therefore, after calculating the training loss of the sample image training, gradient back propagation can be performed according to the training loss to continue training the student network and the teacher network until the preset training end condition is met, and the trained student network is used as the obstacle detection model. Finally, the obstacle detection model is deployed in the smart home device 10 to perform real-time obstacle detection.

[0126] Among them, the preset training end condition is used to determine when to stop training, for example, it can include but is not limited to reaching a preset number of iterations, etc., or, in some other embodiments, it can also be that both the preset threshold and the number of iterations are satisfied, etc., which is not specifically limited here.

[0127] The obstacle detection model acquisition method of the embodiment of the present application uses a lightweight target detection network to construct a student network and a teacher network, and performs multimodal input on the teacher network to combine image information and text information to more comprehensively and effectively obtain the features of the image to be tested, which can improve the student network with a small number of parameters to perceive the environment, thereby improving its image recognition ability. Since the number of parameters of the student network is small, the required computing resources and deployment costs are low, so it can be well applied to smart home devices 10 with limited computing power such as sweepers.

[0128] Figure 7 1 is a flow chart of an obstacle detection method according to an embodiment of the present application. Exemplarily, the obstacle detection method includes:

[0129] S310, obtaining an image to be tested that is captured in real time by the smart home device 10. For example, the image to be tested may be an image of the front environment captured in real time by the smart home device 10 when performing a task.

[0130] S320: Input the image to be tested into the obstacle detection model for processing to obtain an obstacle detection result.

[0131] The structure and specific acquisition method of the obstacle detection model can be found in the description of the above embodiment, which will not be described again here. The optional items in the above embodiment are also applicable to this embodiment, so they will not be described again here.

[0132] Regarding the role of the obstacle detection result, for example, it can be further used for the smart home device 10 to perform operations such as obstacle avoidance, which is not limited here.

[0133] Figure 8 1 is a schematic diagram of a structure of an obstacle detection model acquisition device 100 according to an embodiment of the present application. Exemplarily, the obstacle detection model acquisition device 100 includes:

[0134] The network construction module 110 is used to construct a teacher network and a student network based on the target detection network.

[0135] The student network module 120 is used to process the sample images collected by the smart home device 10 input into the student network to obtain the student features output by the student network at different network layers.

[0136] The text information acquisition module 130 is used to acquire text information of the sample image.

[0137] The teacher network module 140 is used to process the sample images and text information input into the teacher network to obtain the teacher features output by the teacher network at different network layers.

[0138] The network training module 150 is used to calculate the network training loss according to the teacher characteristics and the student characteristics, so as to adjust the network parameters by using the training loss, and use the student network after the training as the obstacle detection model.

[0139] It can be understood that the device of this embodiment corresponds to the obstacle detection model acquisition method of the above embodiment, and the options in the above embodiment are also applicable to this embodiment, so they will not be described repeatedly here.

[0140] The present application also provides a computer-readable storage medium for storing a computer program used in the above-mentioned smart home device 10. For example, the computer-readable storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0141] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of boxes in the structure diagram and / or the flow diagram, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] In addition, the functional modules or units in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0143] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application.

[0144] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for obtaining an obstacle detection model, characterized in that: include: Build the teacher network and student network based on the target detection network; Inputting sample images collected by smart home devices into the student network for processing, and obtaining student features output by the student network at different network layers; Acquire text information of the sample image, input the sample image and the text information into the teacher network for processing, and obtain teacher features output by the teacher network at different network layers; The network training loss is calculated according to the teacher characteristics and the student characteristics, so as to adjust the network parameters by using the training loss, and the student network after the training is used as an obstacle detection model.

2. The obstacle detection model acquisition method according to claim 1, characterized in that: The teacher network and the student network both include: an image feature extraction layer, a feature fusion layer and a prediction output layer connected in sequence; The student features include: image fusion features output by the feature fusion layer in the student network and first prediction features output by the prediction output layer; The teacher features include: image-text fusion features output by the feature fusion layer in the teacher network and second prediction features output by the prediction output layer.

3. The obstacle detection model acquisition method according to claim 2, characterized in that: The training loss includes: a prediction loss of each of the teacher network and the student network, and a distillation loss between the teacher network and the student network, wherein the distillation loss includes an intermediate feature alignment loss and a prediction feature alignment loss; The calculating the network training loss according to the teacher characteristics and the student characteristics includes: According to the first prediction feature and the second prediction feature, respectively calculating the prediction loss of the teacher network and the student network for the sample image, and calculating the prediction feature alignment loss between the teacher network and the student network; The intermediate feature alignment loss between the teacher network and the student network is calculated based on the image-text fusion feature and the image fusion feature.

4. The obstacle detection model acquisition method according to claim 3, characterized in that: The training loss is obtained by weighted summing the prediction loss of the teacher network, the prediction loss of the student network, and the distillation loss; The distillation loss is obtained by weighted summing the intermediate feature alignment loss and the predicted feature alignment loss.

5. The obstacle detection model acquisition method according to claim 3, characterized in that: The calculating the prediction feature alignment loss between the teacher network and the student network comprises: A similarity between the first prediction feature and the second prediction feature is calculated, and the prediction feature alignment loss is calculated based on the similarity.

6. The obstacle detection model acquisition method according to claim 3, characterized in that: The calculating the intermediate feature alignment loss between the teacher network and the student network comprises: A cross entropy loss between the image-text fusion feature and the image fusion feature is calculated as the intermediate feature alignment loss.

7. The obstacle detection model acquisition method according to claim 2, characterized in that: The sample images collected by the smart home device are input into the student network for processing to obtain the student features output by the student network at different network layers, including: Using the image feature extraction layer in the student network to extract features from sample images collected by the smart home device, to obtain a first image feature; Using the feature fusion layer to perform feature fusion on the first image features to obtain the image fusion features; The image fusion feature is predicted using the prediction output layer to obtain the first prediction feature output by the student network.

8. The obstacle detection model acquisition method according to claim 2, characterized in that: The teacher network also includes a text feature extraction layer, wherein the sample image and the text information are input into the teacher network for processing to obtain teacher features output by the teacher network at different network layers, including: Using the image feature extraction layer and the text feature extraction layer in the teacher network to extract features from the sample image and the text information respectively, to obtain second image features and text features; Using the feature fusion layer to fuse the second image feature and the text feature to obtain the image-text fusion feature; The image-text fusion feature is predicted using the prediction output layer to obtain the second prediction feature output by the teacher network.

9. The obstacle detection model acquisition method according to claim 2, characterized in that: The teacher network and the student network are both constructed using the target detection network model of the YOLO series; The network model of the yolo series includes Backbone network, Neck network and Head network, which serve as the image feature extraction layer, the feature fusion layer and the prediction output layer respectively; Among them, the network parameter amount in the teacher network is greater than the network parameter amount in the student network.

10. The obstacle detection model acquisition method according to claim 2, characterized in that: The obtaining of text information of the sample image includes: Input the sample image into the large language model to generate corresponding image description text; use the text feature extraction model to extract features from the image description text to obtain text features; the text features are used to input into the feature fusion layer in the teacher network; Alternatively, when the teacher network also includes a text feature extraction layer, the sample image is input into a large language model to generate a corresponding image description text; the image description text is used to be input into the text feature extraction layer in the teacher network for text feature extraction.

11. The obstacle detection model acquisition method according to claim 10, characterized in that: The text feature extraction model or the text feature extraction layer is constructed using a pre-trained language model.

12. An obstacle detection method, characterized in that: include: Obtain the image to be tested collected by the smart home device in real time; The image to be tested is input into an obstacle detection model for processing to obtain an obstacle detection result; wherein the obstacle detection model is obtained by using the model acquisition method described in any one of claims 1 to 11.

13. An obstacle detection model acquisition device, characterized in that: include: A network construction module is used to construct a teacher network and a student network based on the target detection network; A student network module is used to process sample images collected by smart home devices input into the student network to obtain student features output by the student network at different network layers; A text information acquisition module, used to acquire text information of the sample image; A teacher network module, used for processing the sample image and the text information input into the teacher network to obtain teacher features output by the teacher network at different network layers; The network training module is used to calculate the network training loss according to the teacher characteristics and the student characteristics, so as to adjust the network parameters by using the training loss, and use the student network after the training as an obstacle detection model.

14. A smart home device, characterized in that: The smart home device comprises a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the method according to any one of claims 1 to 12.

15. A readable storage medium, characterized in that: The computer program is stored therein, and when the computer program is executed, the method according to any one of claims 1 to 12 is implemented.

Images