Model training methods, image detection methods, electronic devices and storage media

Abstract

This specification relates to a model training method, an electronic device for an image detection method, and a storage medium. The model training method includes: inputting training image samples, each containing at least one object, into a constructed multi-object image detection model to obtain object information for each object in the training image samples; the multi-object image detection model includes at least two object recognition network branches; determining the branch loss of each object recognition network branch based on preset object information labels and the object information of each object in the training image samples; adjusting the loss weight coefficients of the object recognition network branches based on the source attributes of the preset object information labels; and determining the loss of the multi-object image detection model based on the branch loss and the adjusted loss weight coefficients to obtain a target multi-object image detection model. The embodiments in this specification can quickly complete the detection of multi-object images, thereby improving detection speed.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to a model training method, an image detection method, a mobile robot, and an electronic device. Background Technology

[0002] Robotic vacuum cleaners are increasingly being used in people's lives because of their ability to clean automatically. To achieve better cleaning results, robotic vacuum cleaners need to identify dirt, obstacles, and other contaminants on the floor for targeted cleaning. However, existing recognition models suffer from low recognition efficiency, which further leads to problems such as reduced detection efficiency and poor cleaning performance. Summary of the Invention

[0003] This specification aims to address, at least to some extent, one of the technical problems in the related art. To this end, one objective of this specification is to propose a model training method that enables a mobile robot to perform multi-task image detection using an image detection model, thereby improving the detection speed of the mobile robot.

[0004] The second objective of this specification is to propose an image detection method.

[0005] The third objective of this specification is to propose a mobile robot.

[0006] The fourth objective of this specification is to provide an electronic device.

[0007] The fifth objective of this specification is to provide a computer storage medium.

[0008] To achieve the above objectives, the first aspect of this specification proposes a model training method. This method involves inputting training image samples, including at least one object, into a constructed multi-object image detection model to obtain object information for each object in the training image samples. The multi-object image detection model includes at least two object recognition network branches, which are used to detect object information of objects in the input image. The method determines the branch loss of each object recognition network branch based on preset object information labels and the object information of each object in the training image samples. It adjusts the loss weight coefficients of the object recognition network branches based on the source attributes of the preset object information labels. The loss of the multi-object image detection model is determined based on the branch loss of the object recognition network branches and the adjusted loss weight coefficients, so as to satisfy the model training stopping condition and obtain the target multi-object image detection model. According to the model training method of this specification, a multi-object image detection model including at least two object recognition network branches is constructed, and the multi-object image detection model is trained using training image samples. The branch loss of each object recognition network branch is determined based on the object information of each object in the training image samples detected by the multi-object image detection model and the preset object information of the objects in the training image samples. The loss weight coefficients of the branch loss of the object recognition network are adjusted based on the preset object information label generation method. The loss of the multi-object image detection model is determined according to the branch loss and the adjusted loss weight coefficients of the branch loss. The relevant parameters of the model are adjusted according to the loss, ultimately resulting in a target multi-object image detection model with a detection accuracy reaching a preset precision threshold. The embodiments in this specification enable mobile robots to use the target multi-object image detection model to detect multiple target objects in an image, improving the speed of indoor detection by mobile robots.

[0009] In some embodiments of this specification, adjusting the loss weight coefficient of the object recognition network branch according to the source attribute of the preset object information tag includes: increasing the loss weight coefficient of the object recognition network branch that detects the object corresponding to the preset object information tag when the source attribute of the preset object information tag is a first preset attribute; or decreasing the loss weight coefficient of the object recognition network branch that detects the object corresponding to the preset object information tag when the source attribute of the preset object information tag is a second preset attribute; wherein the confidence level of the first preset attribute is greater than the confidence level of the second preset attribute.

[0010] In some embodiments of this specification, at least two object recognition network branches include a first object recognition network branch for detecting a first type of object and a second object recognition network branch for detecting a second type of object. Adjusting the loss weight coefficients of the object recognition network branches based on the source attributes of preset object information tags includes: if the preset object information tag corresponding to the second type of object is detected by an independent object recognition model corresponding to the second type of object; based on the source attributes of the preset object information tag corresponding to the second type of object, decreasing the loss weight coefficient of the second object recognition network branch and / or increasing the loss weight coefficient of the first object recognition network branch; or, if the preset object information tag corresponding to the first type of object is obtained through manual annotation; based on the source attributes of the preset object information tag corresponding to the first type of object, decreasing the loss weight coefficient of the second object recognition network branch and / or increasing the loss weight coefficient of the first object recognition network branch.

[0011] In some embodiments of this specification, at least two object recognition networks include a job scene classification network branch for job scene classification. The loss weight coefficient of the object recognition network branch is adjusted according to the source attribute of the preset object information label, including: when the preset object information label corresponding to the job scene object in the training image sample indicates that the job scene category in the training image sample cannot be clearly labeled, the preset coefficient threshold is set as the loss weight coefficient of the classification model for image scene classification.

[0012] In some embodiments of this specification, the preset coefficient threshold is set to 0.

[0013] In some embodiments of this specification, training image samples including at least one object are input into a constructed multi-object image detection model to obtain object information of each object in the training image samples, including: determining feature maps of the training image samples by performing convolution processing on the training image samples; inputting the feature maps into at least two object recognition network branches according to preset rules; and the object recognition network branches respectively detecting the object information of the objects included in the input feature maps.

[0014] In some embodiments of this specification, the feature map is input into at least two object recognition network branches according to preset rules, including: inputting the feature map into one of the object recognition network branches of the multi-object image detection model, extracting the portion of the feature map from which no object is detected to obtain a local feature map, and inputting the local feature map into other object recognition network branches of the multi-object image detection model; or, extracting a specified portion from the feature map to obtain at least one local feature map, and inputting at least one local feature map and the feature map respectively into each object recognition network branch of the multi-object image detection model according to a preset correspondence.

[0015] In some embodiments of this specification, the multi-object image detection model further includes a network backbone for performing convolutional processing on the image; at least two object recognition network branches are connected to the network backbone; the at least two object recognition network branches include a first branch for recognizing a first category of objects and a second branch for recognizing a second category of objects; the size of the second category of objects is smaller than the size of the first category of objects; inputting the feature map into the at least two object recognition network branches according to a preset rule includes: inputting the feature map into the first branch; extracting a specified portion from the feature map to obtain a local feature map, and inputting the local feature map into the second branch.

[0016] In some embodiments of this specification, the multi-object image detection model includes at least two object recognition network branches from the following: a dirt detection network branch, an obstacle detection network branch, a furniture detection network branch, and a work scene classification network branch.

[0017] To achieve the above objectives, a second aspect of this specification provides an image detection method, which includes: acquiring an image to be detected; inputting the image to be detected into a target multi-object image detection model obtained through any of the above embodiments for detection, and obtaining object information of the target objects included in the image to be detected.

[0018] According to the image detection method of this application, an image to be detected is acquired, and the image to be detected is input into the trained target multi-object image detection model for detection to obtain the target objects included in the image to be detected. The embodiments of this specification enable a mobile robot to use the target multi-object image detection model to detect multiple targets in an image, improving the speed of indoor detection by the mobile robot.

[0019] To achieve the above objectives, a third aspect of this specification provides a mobile robot, which includes: a sensing device, a memory, a processor, and a target detection program for the mobile robot stored in the memory. The image detection program for the mobile robot is executed by the processor to implement the steps of the image detection method as described in the second aspect of the specification.

[0020] According to the embodiments of this specification, when the image detection program of the mobile robot is executed by the processor, the mobile robot can use a target multi-object image detection model to complete the detection of multiple target objects in the image, thereby improving the indoor detection speed of the sweeping robot.

[0021] To achieve the above objectives, a fourth aspect of this specification provides an electronic device, including a memory, a processor, and a model training program stored in the memory and executable on the processor. When the processor executes the model training program, it implements the model training method as described in any of the first aspect embodiments.

[0022] According to the embodiments of this specification, when the processor executes the model training program, the mobile robot can use the target multi-object image detection model to complete the detection of multiple target objects in the image, thereby improving the indoor detection speed of the sweeping robot.

[0023] To achieve the above objectives, a fifth aspect of this specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a model training method as described in any embodiment of the first aspect or an image detection method as described in the second aspect.

[0024] According to embodiments of this specification, a computer-readable storage medium, when a model training program or an image detection program is executed by a processor, enables a mobile robot to use a target multi-object image detection model to detect multiple target objects in an image, thereby improving the indoor detection speed of the sweeping robot.

[0025] Additional aspects and advantages of this specification will be set forth in part in the description which follows, and in part will be obvious from the description or may be learned by practice of this specification. Attached Figure Description

[0026] Figure 1 This is an application scenario diagram of the model training method according to the embodiments of this specification.

[0027] Figure 2 This is a flowchart of a model training method according to an embodiment of this specification.

[0028] Figure 3a This is a schematic diagram of model training in related technologies.

[0029] Figure 3b This is a schematic diagram illustrating the generation of training image samples in one embodiment of this specification.

[0030] Figure 4 This is a schematic diagram of the network branch structure in one embodiment of this specification.

[0031] Figure 5 This is a flowchart of an image detection method according to an embodiment of this specification.

[0032] Figure 6 This is a structural block diagram of a mobile robot according to an embodiment of this specification.

[0033] Figure 7 This is a structural block diagram of an electronic device according to an embodiment of this specification. Detailed Implementation

[0034] The embodiments of this specification are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this specification, and should not be construed as limiting this specification.

[0035] In daily life, robotic vacuum cleaners are increasingly used due to their ability to clean automatically. When working, a robotic vacuum cleaner can photograph its surrounding environment, detect the images, determine its working scenario, and identify areas of dirt, obstacles, and furniture within the environment. This allows the robotic vacuum cleaner to avoid obstacles and furniture while cleaning, selecting the appropriate working mode based on the scenario. However, current methods require multiple detection models to identify obstacles and furniture, which is time-consuming and cumbersome, resulting in redundant network computing overhead and reducing the detection speed of the robotic vacuum cleaner.

[0036] Therefore, based on the aforementioned related technologies, this application proposes a model training method, an image detection method, an electronic device (including a mobile robot), and a computer-readable storage medium, which enables the mobile robot to use a trained multi-object image detection model to detect multiple targets in an image, thereby improving the detection speed of the sweeping robot.

[0037] To achieve ground dirt detection, obstacle and furniture detection, and scene recognition, this invention trains three corresponding models to detect and recognize ground dirt, obstacles, furniture, and scene classification in images, respectively. Specifically, the three models are a ground dirt detection model, an obstacle and furniture detection model, and a scene recognition model. The ground dirt detection model is specifically used to detect and identify dirty objects in the image, outputting the location and dirt label of each dirty object in the image; the obstacle and furniture detection model is used to detect and identify obstacle and furniture objects in the image, outputting the location and obstacle label of the obstacle object and the location and furniture label of the furniture object; the scene recognition model is used to identify the scene category of the image (the working scene of the mobile robot), outputting scene category labels. When these three models are trained individually, the sample images required for each model are different. The ground dirt detection model requires manually labeled sample images of ground dirt, which are stored separately in image library 1. The obstacle and furniture detection model requires manually labeled sample images of obstacles and furniture, which are stored separately in image library 2. The scene recognition model requires manually labeled sample images of scene categories, which are stored in image library 3. After these three models are trained separately, they can respectively achieve ground dirt detection, obstacle and furniture detection, and scene category recognition.

[0038] After training three models for identifying dirt on the ground, obstacles, and furniture, as well as scene classification, according to the logic of existing technologies, if information on dirt on the ground, obstacles, furniture, and scene category in an image is obtained, the image must be sequentially input into these three models for processing. This method of multiple models detecting separately, as used in existing technologies, is time-consuming and cumbersome. Moreover, even with the furniture detection model and scene recognition model provided in this invention, existing technologies cannot utilize the gain of furniture detection information for scene classification (based on the identified furniture information, further scene classification is easier) when the scene recognition model identifies scene categories, resulting in certain redundant network computing overhead, thereby reducing the detection speed of the robot vacuum cleaner.

[0039] Specifically, the model training method provided in the embodiments of this specification can be applied to... Figure 1The example application scenario diagram shows a robotic vacuum cleaner 102, a first server 104, and a second server 106. The first server 104 has been trained with models for detecting dirt on the floor, obstacles and furniture, and scene recognition, and has deployed image libraries (Image Library 1, Image Library 2, Image Library 3). Based on the images stored in the image libraries, the first server 104 can determine preset object information labels for various objects included in the images. For example, an image with manually labeled preset dirt information labels is input into the obstacle and furniture detection model, which detects obstacle and furniture objects, obtaining the preset obstacle information labels and preset furniture information labels corresponding to the obstacle and furniture objects in the image. Next, the image is input into the scene recognition model, which detects the corresponding work scene category, obtaining the preset scene information labels corresponding to the work scene objects. Through the above image processing, preset object information labels for various objects included in the image can be obtained, such as preset dirt information labels, preset obstacle information labels, preset furniture information labels, and preset scene information labels. The preset object information labels record the coordinate position of the object in the image and / or the object's type information. For example, a preset obstacle information label for an obstacle object in an image records the coordinate position of a certain obstacle object in the image and the category of the object, "obstacle".

[0040] The second server 106 is used to build an initial multi-object image detection model based on existing ground dirt detection models, obstacle and furniture detection models, and work scene classification network models, or according to constructed dirt detection network branches, obstacle detection network branches, furniture detection network branches, and work scene classification network branches. The multi-object image detection model includes at least two parallel network branches. It is understood that this example introduces the first and second servers for ease of explanation, but other examples may only have one server, and the technical solution described in this specification can be implemented based on a single server.

[0041] In some embodiments, the multi-object image detection model includes four object recognition network branches: a dirt detection network branch, an obstacle detection network branch, a furniture detection network branch, and a work scene classification network branch. Specifically, the dirt detection network branch is dedicated to detecting and identifying dirty objects in the image, outputting the location and dirt label of each dirty object; the obstacle detection network branch is used to detect and identify obstacle objects in the image, outputting the location and obstacle label of all obstacle objects included in the image; the furniture detection network branch is used to detect and identify furniture objects in the image, outputting the location and furniture label of all furniture objects included in the image; and the work scene classification network branch is used to identify the scene category of the image (the work scene of the mobile robot), outputting a scene category label.

[0042] Images in the image library can be used as training image samples to train an initial multi-object image detection model. In this scenario example, since the preset object information labels for the various objects used to train the image samples can be obtained by manual annotation or by model detection, the preset object information labels also include the source attribute of the object information. If the preset object information labels are obtained by manual annotation, then the source attribute of the preset object information labels is manual annotation (first preset attribute); if the preset object information labels are obtained by model detection, then the source attribute of the preset object information labels is machine annotation (second preset attribute).

[0043] Manual annotation involves manually circling objects on an image and labeling them with information such as furniture (doors, beds), dirt (hair, scraps of paper), work scene (living room, bedroom), and obstacles (boxes, trash cans), which generates pre-defined object information labels. Machine annotation, on the other hand, involves inputting the image into a model / branch, such as a work scene classification network branch, which can detect the corresponding work scene category and obtain pre-defined scene information labels for the work scene objects. This process is machine annotation.

[0044] After completing sample construction and building the multi-object image detection model, training of the image detection model can begin. Specifically, the second server 106 inputs training image samples, each containing at least one object, into the built multi-object image detection model to obtain object information for each object in the training image samples. The multi-object image detection model includes at least two object recognition network branches, which are used to detect object information in the input images. Based on the preset object information labels of the objects in the training image samples and the object information of each object in the training image samples, the branch loss of the object recognition network branches is determined. The loss weight coefficients of the object recognition network branches are adjusted according to the source attributes of the preset object information labels. Based on the branch loss of the object recognition network branches and the adjusted loss weight coefficients, the loss of the multi-object image detection model is determined to satisfy the model training stopping condition, thus obtaining the target multi-object image detection model. The target multi-object image detection model trained by the second server 106 can be deployed in the sweeping robot 102. The robotic vacuum cleaner 102 can acquire an image to be detected through an image acquisition device, input the image to be detected into a target multi-object image detection model for detection, and obtain object information of at least one object in the image to be detected, including dirt, furniture, obstacles, and the working scene where the robotic vacuum cleaner is located.

[0045] Figure 2 This is a flowchart of the model training method in the embodiments of this specification, such as... Figure 2 As shown, the model training method includes the following steps:

[0046] S210, input training image samples including at least one object into the constructed multi-object image detection model to obtain object information of each object in the training image samples.

[0047] The multi-object image detection model includes at least two object recognition network branches, which are used to detect object information of objects in the input image.

[0048] In the embodiments of this specification, a multi-object image detection model is built to detect and identify ground dirt, obstacles, furniture, and scene categories in an image within a single model. The built multi-object image detection model includes at least two parallel object recognition network branches, each of which can be used to detect one or more objects in the image, including ground dirt, obstacles, furniture, and scene categories. In the embodiments of this specification, the multi-object image detection model can be built and generated based on pre-trained ground dirt detection models, obstacle and furniture detection models, and work scene classification models from related technologies. Before construction, these models can run independently; after construction, these independent models become one object recognition network branch of the multi-object image detection model. The multi-object image detection model includes at least two parallel object recognition network branches, including a dirt detection network branch for detecting ground dirt, an obstacle detection network branch for detecting obstacles, a furniture detection network branch for detecting furniture, and a work scene classification network branch for scene category recognition.

[0049] In the embodiments of this specification, based on four network structures—dirt detection network branch, obstacle detection network branch, furniture detection network branch, and work scene classification network branch—the multi-object image detection model can be constructed as a model capable of recognizing two, three, or four types of objects. If the multi-object image detection model can recognize two types of objects, then the two parallel object recognition network branches can be any two of the following network structures: dirt detection network branch, obstacle detection network branch, furniture detection network branch, and work scene classification network branch. If the multi-object image detection model can recognize four types of objects, then the two parallel object recognition network branches can be constructed from the four network structures: dirt detection network branch, obstacle detection network branch, furniture detection network branch, and work scene classification network branch. For example, the dirt detection network branch and the obstacle detection network branch can also be combined into one object recognition network branch, and the furniture detection network branch and the work scene classification network branch can be combined into another object recognition network branch.

[0050] The training image samples include four categories of objects: obstacle objects, dirty floor objects, furniture objects, and work scene objects. Each object category can include one or more more specific objects. For example, obstacles can refer to indoor carpets, toys, corners, etc., and scene categories can be divided into living room, bedroom, bathroom, and kitchen, etc. The specific settings can be configured according to the application scenario of the mobile robot. For example, the training image samples may include multiple dirty floor objects, multiple obstacle objects, multiple furniture objects, and one work scene object.

[0051] Two object recognition networks are used to detect object information in the input image. In the embodiments of this specification, the object information obtained by detecting ground debris, obstacles, and furniture objects in the input image includes the coordinate position and category of the object in the image. The object information obtained by detecting work scene objects in the input image includes the work scene category displayed by the input image. For example, if one of the object recognition network branches can detect furniture objects and work scene objects in the input image, the obtained furniture object information includes the coordinate position of the furniture object in the input image and the category "furniture"; the obtained work scene object information includes the work scene category "kitchen" represented by the work scene object.

[0052] S220, Based on the preset object information labels of objects in the training image samples and the object information of each object in the training image samples, determine the branch loss of the object recognition network branch.

[0053] In the embodiments of this specification, the preset object information labels of objects in the training image samples can be determined before training the multi-object image detection model. The preset object information labels of objects in the training image samples can be obtained manually or by machine annotation. Generally, preset object information labels for objects of the same category are annotated in the same way. For example, the preset object information labels corresponding to dirt-related objects on the ground in the training image samples can be obtained manually, while the preset object information labels for other object categories can be obtained by machine annotation. Because the annotation methods for preset object information labels differ, the preset object information labels also include a source attribute to indicate how the preset object information labels were generated.

[0054] For example, the training image samples can be images from image library 1, image library 2, or image library 3 in the aforementioned related technologies. For example... Figure 3aAs shown, the images in image library 1 only have preset dirt information labels for manually labeled ground dirt objects, which can be used to directly train the aforementioned ground dirt detection model; the images in image library 2 have preset obstacle information labels for manually labeled obstacle objects and / or preset furniture information labels for manually labeled furniture objects, which can be used to directly train the aforementioned obstacle and furniture detection model; the images in image library 3 only have preset scene information labels for manually labeled work scene objects, which can be used to directly train the aforementioned scene recognition model. If the multi-object image detection model of this embodiment can detect four types of objects in the image: ground dirt, obstacles, furniture, and scene categories. Figure 3b As shown, if images from Image Library 1, Image Library 2, and Image Library 3 are used, data annotation and correction are required to obtain training image samples labeled with preset dirt information tags, preset obstacle information tags, preset furniture information tags, and preset scene information tags. For example, if images from Image Library 1 are used as training image samples, and these samples currently only have manually labeled preset ground information tags, then these training image samples can be input into the aforementioned obstacle and furniture detection model to obtain the preset obstacle information tags and preset furniture information tags for the training image samples. Then, they can be input into the work scene classification network branch to obtain the preset scene information tags for the training image samples. Finally, training image samples labeled with preset object information tags for four types of objects can be obtained in this embodiment of the specification. In other implementable methods, if images from Image Library 1 are used as training image samples, the preset obstacle information tags, preset furniture information tags, and preset scene information tags for the training image samples can also be manually labeled.

[0055] When training a model, it is usually necessary to calculate the loss value based on the loss function to adjust the model's parameters, ultimately obtaining a target model with a recognition accuracy reaching a preset threshold. In the embodiments of this specification, the multi-object image detection model includes at least two parallel object recognition network branches, so the branch loss of each object recognition network branch can be calculated separately. Then, the loss value of the multi-object image detection model is calculated based on the branch loss of each object recognition network branch.

[0056] In the embodiments of this specification, the detection of ground dirt objects, obstacle objects, and furniture objects in the training image samples includes object location detection and object category recognition. Therefore, in the object recognition network branch, the object recognition network branch for ground dirt objects, obstacle objects, and furniture objects includes a detection head and a classification head, and the branch loss of the object recognition network branch consists of the detection box loss of the detection head and the classification loss of the classification head.

[0057] The detection of work scene objects in the training image samples includes object category recognition; therefore, the object recognition network branch for work scene objects only includes the classification head. Consequently, the branch loss of the object recognition network branch used to detect work scene objects only includes the classification loss of the classification head.

[0058] Based on the above theory, the loss value of the detection head in the object recognition network branch can be determined based on the detection box loss function, and the loss value of the classification head in the object recognition network branch can be determined based on the classification loss function.

[0059] For example, the GIoU Loss function can be used as the bounding box loss function, and the softmax loss function can be used as the classification loss function. In the field of object detection, the GIoU Loss function calculates the loss by comparing the predicted bounding box (preBBox) with the actual labeled bounding box (ground truth BBox). The larger the overlap between the two boxes, the smaller the loss, and vice versa. The softmax loss function is used to evaluate the similarity between the model's predictions and the true values; the smaller the loss function, the better the model's robustness. The specific calculation methods for the two loss functions will not be described here.

[0060] Based on the preset object information labels of objects in the training image samples and the object information of each object in the training image samples, the branch loss of the object recognition network branch is determined. For example: based on the preset object information labels corresponding to ground dirt objects in the training image samples and the ground dirt object information obtained by the multi-object image detection model from detecting ground dirt objects in the training image samples, the detection loss value and classification loss value of the dirt detection network branch used to detect ground dirt objects in the object recognition network branch are calculated based on the detection box loss function and the classification loss function. Similarly, based on the preset object information labels corresponding to obstacle objects in the training image samples and the obstacle object information obtained by the multi-object image detection model from detecting obstacle objects in the training image samples, the detection loss value and classification loss value of the obstacle detection network branch used to detect obstacle objects in the object recognition network branch are calculated based on the detection box loss function and the classification loss function.

[0061] If an object recognition network branch includes only one object detection network branch for a single object class, then the detection loss and classification loss values ​​corresponding to that object detection network branch constitute the branch loss of that object recognition network branch. If an object recognition network branch is composed of two object detection network branches, then the detection loss and classification loss values ​​of these two object detection network branches constitute the branch loss of that object recognition network branch. Object detection network branches can represent dirt detection network branches, obstacle detection network branches, furniture detection network branches, or work scene classification network branches.

[0062] S230, adjust the loss weight coefficient of the object recognition network branch according to the source attribute of the preset object information label.

[0063] In some cases, the accuracy of the labeled object information varies depending on the generation method of the preset object information labels. It is understandable that the confidence level of preset object information labels annotated manually is higher than that of preset object information labels annotated by machines. If a multi-object image detection model relies more on preset object information labels with higher confidence levels to adjust its parameters during training, the model's recognition accuracy can be improved. In the embodiments of this specification, the source attribute in the preset object information labels can be used to determine the generation method of the preset object information labels. Therefore, the loss weight coefficients of the object recognition network branches can be adjusted according to the source attribute of the preset object information labels.

[0064] Specifically, the loss weight coefficient of the object recognition network branch can be increased or decreased based on the source attribute of the preset object information label. When the object recognition network branch is composed of multiple object detection network branches, the loss weight coefficient of the object recognition network branch includes the loss weight coefficients of the multiple object detection network branches. Therefore, adjusting the loss weight coefficient of the object recognition network branch based on the source attribute of the preset object information label can include: adjusting the loss weight coefficient of the object detection network branch within the object recognition network branch used to detect the corresponding object of the preset object information label based on the preset object information label.

[0065] S240, the loss of the multi-object image detection model is determined based on the branch loss of the object recognition network branch and the adjusted loss weight coefficient, so as to satisfy the model training stopping condition and obtain the target multi-object image detection model.

[0066] The loss function can be used to guide model learning. The model parameters are modified by backpropagation based on the loss function. The purpose of machine learning is to learn a set of parameters so that the predicted value is infinitely close to the true value.

[0067] In the embodiments of this specification, the loss of the multi-object image detection model can be obtained by multiplying the branch loss of the object recognition network branch by its adjusted loss weight coefficient and then summing the results. Since the multi-object image detection model in the embodiments of this specification includes both location detection and category recognition for the input image, the loss of the multi-object image detection model includes bounding box loss and classification loss.

[0068] For example, if the constructed multi-object image detection model includes two parallel object recognition network branches, one of which is composed of a dirt detection network branch and an obstacle detection network branch, and the other is composed of a furniture detection network branch and a work scene classification network branch, then the detection box loss of the multi-object image detection model is obtained by multiplying the detection box loss values ​​of the dirt detection network branch, the obstacle detection network branch, and the furniture detection network branch by their respective adjusted loss weight coefficients and then summing them. The classification loss of the multi-object image detection model is obtained by multiplying the classification loss values ​​of the dirt detection network branch, the obstacle detection network branch, the furniture detection network branch, and the work scene classification network branch by their respective adjusted loss weight coefficients and then summing them.

[0069] The parameters of the multi-object image detection model are adjusted based on the loss of the trained model. This process continues until the loss of the model reaches a preset loss threshold after multiple training rounds. When the loss reaches this threshold, the model's recognition accuracy has reached the accuracy indicated by the model training stopping condition. Once the stopping condition is met, training of the multi-object image detection model is stopped, resulting in the target multi-object image detection model.

[0070] According to the model training method of the embodiments in this specification, a multi-object image detection model capable of detecting multiple object classes is constructed, and the model is trained using training image samples. The loss weight coefficients of the corresponding object recognition network branches are adjusted based on the generation method of preset object information labels in the training image samples. This makes the adjustment of the parameters of the multi-object image detection model more reliant on preset object information labels with higher confidence, thereby improving the detection accuracy and recognition precision of the multi-object image detection model. Furthermore, by functionally integrating independent object detection models from related technologies to construct a single model for application in mobile robots, it facilitates subsequent model maintenance and inference development.

[0071] In some embodiments of this specification, the loss weight coefficient of the object recognition network branch is adjusted according to the source attribute of the preset object information tag. This may include: increasing the loss weight coefficient of the object recognition network branch detecting the object corresponding to the preset object information tag when the source attribute of the preset object information tag is a first preset attribute (manually labeled); or decreasing the loss weight coefficient of the object recognition network branch detecting the object corresponding to the preset object information tag when the source attribute of the preset object information tag is a second preset attribute (machine labeled). Wherein, the confidence level of the first preset attribute (manually labeled) is greater than the confidence level of the second preset attribute (machine labeled).

[0072] In the embodiments of this specification, the confidence level of manually labeled preset object information tags is higher than that of machine-labeled preset object information tags. Therefore, the first preset attribute can be used to indicate that the preset object information tags are generated by manual labeling; the second preset attribute can be used to indicate that the preset object information tags are generated by machine labeling. Specifically, the attribute value of the source attribute of the preset object information tags can be expressed as "manual labeling", "machine labeling", "0", "1", a specific source path, or other representations. The first preset attribute can include attribute values ​​represented by "manual labeling", "0", or a specific source path; the second preset attribute can include attribute values ​​represented by "machine labeling", "1", or a null value. It should be noted that when the training image samples use images from image library 1, image library 2, or image library 3 in the first server, the specific source path of image library 1, image library 2, or image library 3 can be used as the source attribute value of the preset object information tags.

[0073] In the embodiments of this specification, increasing the loss weight coefficient of the object recognition network branch used to detect preset object information labels annotated manually, or decreasing the loss weight coefficient of the object recognition network branch used to detect preset object information labels annotated by machine, can both achieve the goal of making the multi-object image detection model more confident in the preset object information labels. Specifically, if the object recognition network branch is composed of multiple object detection network branches, and the source attributes of the preset object information labels of the objects that can be detected by the multiple object detection network branches are consistent, then the loss weight coefficients of the multiple object detection network branches are increased or decreased according to the source attributes. If the object recognition network branch is composed of multiple object detection network branches, but the source attributes of the preset object information labels of the objects that can be detected by the multiple object detection network branches are inconsistent, then the loss weight coefficient of the object detection network branch used to detect objects corresponding to preset object information labels with a first preset attribute as the source attribute is increased, and the loss weight coefficient of the object detection network branch used to detect objects corresponding to preset object information labels with a second preset attribute as the source attribute is decreased.

[0074] In some embodiments of this specification, at least two object recognition network branches include a first object recognition network branch for detecting a first type of object and a second object recognition network branch for detecting a second type of object. The loss weight coefficients of the object recognition network branches are adjusted based on the source attributes of preset object information tags. This may include: if the preset object information tag corresponding to the second type of object is obtained by an independent object recognition model corresponding to the second type of object; based on the source attributes of the preset object information tag corresponding to the second type of object, decreasing the loss weight coefficient of the second object recognition network branch, and / or increasing the loss weight coefficient of the first object recognition network branch. Alternatively, if the preset object information tag corresponding to the first type of object is obtained through manual annotation; based on the source attributes of the preset object information tag corresponding to the first type of object, decreasing the loss weight coefficient of the second object recognition network branch, and / or increasing the loss weight coefficient of the first object recognition network branch.

[0075] Specifically, the independent object recognition model can be any of the aforementioned ground dirt detection model, obstacle and furniture detection model, and work scene classification network branch. The preset object information labels corresponding to the first type of object are obtained through manual annotation. The preset object information labels corresponding to the second type of object are obtained through machine annotation, i.e., detected by the independent object recognition model corresponding to the second type of object. Adjusting the loss weight coefficient of the object recognition network branch based on the source attribute of the preset object information labels can include reducing the loss weight coefficient of the second object recognition network branch or increasing the loss weight coefficient of the first object recognition network branch based on either the source attribute of the preset object information labels corresponding to the second type of object or the source attribute of the preset object information labels corresponding to the first type of object; or both reducing the loss weight coefficient of the second object recognition network branch and increasing the loss weight coefficient of the first object recognition network branch.

[0076] It's understandable that multi-object image detection models include at least two object recognition network branches. Taking a multi-object image detection model with two object recognition network branches as an example, in this case, the two object recognition network branches in the multi-object image detection model can simultaneously identify two objects in the input image. For independent object recognition models, which can refer to a single object recognition network branch, these models can only output one or more objects from the input image. Here, "object" does not refer to specific objects like beds or tables, but rather to furniture objects, including beds and tables. The "one object" mentioned here refers to identifying object information in the input image and labeling the identified objects. Of course, both multi-object image detection models and independent object recognition models can have instances where objects are not identified or are missed.

[0077] In some embodiments of this specification, at least two object recognition networks include a job scene classification network branch for job scene classification, and the loss weight coefficients of the object recognition network branch are adjusted according to the source attributes of preset object information labels. This includes setting a preset coefficient threshold as the loss weight coefficient of the job scene classification network branch when the preset object information labels corresponding to job scene objects in the training image samples indicate that the job scene category in the training image samples cannot be clearly labeled.

[0078] In some cases, due to the shooting angle of certain images, models or human annotators may be unable to accurately identify the job scene category of an image. For example, if an image is taken from a location where both a bathroom and a living room are in the same viewpoint, resulting in a partially visible bathroom and partially visible living room, then the job scene category cannot be distinguished. Using such images as training samples in an image detection model may negatively impact the model's training results. Therefore, it is necessary to adjust the loss weight coefficients of the job scene classification network branch used for detecting job scenes in the image detection model for these training image samples to reduce the poor model training performance caused by scene blur.

[0079] Specifically, at least two object recognition networks include a task scene classification network branch for classifying task scenes, which is used to identify task scenes in training image samples. When the preset object information label corresponding to the task scene object in the training image sample cannot explicitly label the task scene category, a preset coefficient threshold is set as the loss weight coefficient of the task scene classification network branch. The situation where the task scene category in the training image sample cannot be explicitly labeled includes at least the following two cases: First, the preset scene information label of the task scene object in the training image sample is labeled by a human annotator. When the human annotator cannot identify the specific scene of the image, a preset symbol can be used as the scene information for the preset scene information label; Second, the preset scene information label of the training image sample is obtained by machine annotation. When the model identifies that the training image sample includes two or more scene categories, the scene information for the preset scene information label of the training image sample is labeled as "unrecognizable" or another preset symbol.

[0080] In one possible implementation, the preset coefficient threshold can be set to be less than the loss weight coefficient corresponding to the task scenario classification network branch when no adjustment is made. In some embodiments of this specification, the preset coefficient threshold can be 0.

[0081] In some embodiments of this specification, training image samples including at least one object are input into a constructed multi-object image detection model to obtain object information of each object in the training image samples, including:

[0082] The training image samples are convolutionally processed to determine feature maps. These feature maps are then input into at least two object recognition network branches according to preset rules. Each object recognition network branch detects object information contained in the input feature maps.

[0083] In some cases, the size of the training image samples may not be suitable for the processing of multi-object image detection models, and the range and size of dirt, obstacles and furniture appearing in the training image samples vary greatly. Therefore, the training image samples can be preprocessed based on the commonalities between objects, thereby reducing computational overhead and improving detection speed.

[0084] Specifically, after inputting training image samples into the multi-object image detection model, before aligning and performing object detection, the training image samples can be convolved. Convolution processing yields feature maps of the training image samples, identifying all objects within them, facilitating subsequent location detection and category recognition for each object. Since furniture objects are scattered throughout the entire training image sample, while dirty objects may only appear in the lower half of the image, input rules are pre-defined. The feature maps are input into at least two object recognition network branches according to these pre-defined rules. The object recognition network branches detect the object information of the objects included in the input feature maps. For example, if one object recognition network branch is used to detect the object information of dirty objects and obstacle objects in the input image, this branch will detect each object included in the input feature map, identifying the dirty objects and obstacle objects, and determining the object information of the two types of objects.

[0085] In some embodiments of this specification, the feature map is input into at least two object recognition network branches according to preset rules, including: inputting the feature map into one of the object recognition network branches of the multi-object image detection model, extracting the portion of the feature map from which no object is detected to obtain a local feature map, and inputting the local feature map into other object recognition network branches of the multi-object image detection model; or, extracting a specified portion from the feature map to obtain at least one local feature map, and inputting at least one local feature map and the feature map respectively into each object recognition network branch of the multi-object image detection model according to a preset correspondence.

[0086] Specifically, there are at least two preset rules for inputting feature maps into the at least two object recognition network branches according to preset rules. The first is to input the feature map into one object recognition network branch of the multi-object image detection model, extract the portion of the feature map from which no objects are detected to obtain a local feature map, and then input the local feature map into other object recognition network branches of the multi-object image detection model. In some other examples, after inputting the local feature map into other object recognition network branches of the multi-object image detection model, the steps of repeatedly extracting the portion of the feature map from which no objects are detected to obtain a local feature map and inputting the local feature map into other object recognition network branches are further included until all the object recognition network branches have detected the input image. The second is to extract at least one local feature map from a specified portion of the feature map, and input the at least one local feature map and the feature map respectively into each object recognition network branch of the multi-object image detection model according to a preset correspondence. The preset correspondence here can be based on different cropping locations, inputting the corresponding local feature maps obtained from the cropping to the object recognition network branches. For example, the feature map (global feature map) is input to the task scene classification network branch and / or obstacle detection network branch, while the cropped local feature map, such as a local feature map mainly showing the ground, is input to the dirt detection network branch for detection. In some embodiments of this specification, the multi-object image detection model also includes a network backbone for performing convolution processing on the image. At least two object recognition network branches are connected to the network backbone. The at least two object recognition network branches include a first branch for recognizing a first category of objects and a second branch for recognizing a second category of objects. The size of the second category of objects is smaller than the size of the first category of objects. Inputting the feature map into the at least two object recognition network branches according to preset rules can include: inputting the feature map into the first branch; cropping a specified portion from the feature map to obtain a local feature map, and inputting the local feature map into the second branch.

[0087] Specifically, refer to Figure 4The multi-object image detection model may include a network backbone for convolutional processing of training image samples, and at least two parallel object recognition network branches connected to the network backbone. Based on the characteristics of dirt objects, obstacles, furniture objects, and work scene objects in terms of size, location, and required recognition information, objects with many common features can be detected and recognized in the same branch. In some embodiments of this specification, dirt and obstacles are small in size and are concentrated in the lower half of the training image samples, i.e., the ground portion of the training image samples. Therefore, dirt and obstacles can be recognized as second-category objects in the second branch. Furniture recognition has a significant gain for work scene recognition; therefore, furniture objects and work scenes can be recognized as first-category objects in the first branch.

[0088] The backbone of the multi-object image detection model can adopt a network structure with three convolutional layers, such as the streamlined MobileNet, the computationally accurate ShuffleNet, ResNet, Regvgg, and DarNet network structures, all of which can serve as the backbone of the multi-object image detection model in the embodiments of this specification. In practical applications, the backbone can be selected based on hardware computing power and the accuracy of the test data. At least two parallel object recognition network branches connected to the backbone in the multi-object image detection model include multiple detection heads and classification heads. For example, if the first branch is used to detect and recognize furniture objects and work scenes, then the first branch includes a detection head and a classification head for detecting and recognizing furniture, and a classification head for recognizing work scenes. If the second branch is used to detect and recognize dirt objects on the ground and obstacle objects, then the second branch includes a detection head and a classification head for detecting and recognizing dirt objects on the ground, and a detection head and a classification head for detecting and recognizing obstacles. The detection head can obtain the position and coordinates of the object in the training image samples, and the classification head can identify the category of the object.

[0089] Before performing convolutional processing on the training image samples through the network backbone, the input training image samples are usually resized to a preset size, such as 360*360 pixels. When defining the convolutional layers in the network backbone, parameters such as kernel size, stride, and padding are typically set. These parameters ensure that the convolutional kernel can scan from the first pixel to the last pixel of the training image sample. Different parameters will result in different sizes of the final output feature map, calculated using the following formula: Where M is the width of the output feature map and N is the width of the training image samples before convolution.

[0090] In the embodiments of this specification, after convolutional processing of the training image samples through the network backbone, a global feature map of the training image samples can be obtained. To accommodate the subsequent parallel detection of four objects by two branches, the kernel size of all three convolutional layers in the network backbone can be set to 3, the feature map padding width can be set to 2, the stride of one convolutional layer can be set to 2, and the stride of the other two convolutional layers can be set to 1, ultimately resulting in a global feature map with a width approximately half of the aforementioned preset specifications. For example, if the training image samples are adjusted to a size of 360*360 and input into the network backbone, after three convolutional layers in the network backbone, a global feature map of size 180.5*180.5 can be obtained. This feature map includes all objects in the training image samples.

[0091] refer to Figure 4 Since the first branch is used to detect and identify the first category of objects, namely furniture and work scenes, the feature map can be directly input into the first branch. The classification head used in the first branch for furniture detection and identification is used to determine all furniture objects in the feature map, and the detection head is used to determine the position of the furniture objects in the feature map. The classification head used in the first branch for scene category identification is used to determine the work scene of the feature map. Finally, the object information corresponding to the first category of objects can be obtained. The object information corresponding to the first category of objects includes information on multiple furniture objects and information on one work scene object. The furniture object information includes category attributes and position attributes, and the work scene object information includes category attributes.

[0092] Continue to refer to Figure 4 Since the dirt and obstacles in this embodiment are generally located on the indoor floor, and the floor portion is located in the lower half of the feature map, the lower half of the feature map can be extracted as a local feature map. The local feature map is input into the second branch, which then detects and identifies the second category of objects in the local feature map, namely, dirt objects and obstacle objects, to obtain the object information corresponding to the second category of objects. The object information corresponding to the second category of objects may include multiple dirt object information and multiple obstacle object information, both of which include location attributes and category attributes.

[0093] According to the model training method of the embodiments of this specification, preset object information labels of ground dirt, obstacles, furniture, and scene objects in images from related technologies are fused to obtain training image samples and preset object information labels of the objects for a multi-object image detection model. The multi-object image detection model is trained using these training image samples. Common features in the training image samples are effectively extracted from the network backbone of the multi-object image detection model, and data sharing is achieved between parallel object recognition network branches, thus improving the detection speed of the multi-object image detection model. Furthermore, the loss weight coefficients of the object recognition network branches of the multi-object image detection model are adjusted based on the preset object information labels with high credibility in the training image samples, greatly improving the detection accuracy and precision of the model. Integrating multiple detection models from related technologies into a single detection model for application to mobile robots also facilitates subsequent model maintenance and inference development.

[0094] Figure 5 The diagram shown is a flowchart of the image detection method in an embodiment of this specification. Figure 5 As shown, the image detection method includes the following steps:

[0095] S510, acquire the image to be detected.

[0096] S520, the image to be detected is input into the target multi-object image detection model obtained in any of the above embodiments for detection, and the object information of the target objects included in the image to be detected is obtained.

[0097] In the embodiments of this specification, the image to be detected can be captured by a mobile robot while it is moving indoors. Typically, images captured by a mobile robot are 640*360 pixels in size, while the target multi-object image detection model in this embodiment can process images of 360*360 pixels. Therefore, before inputting the image captured by the mobile robot into the target multi-object image detection model for processing, the image size can be adjusted to 360*360 pixels for detection.

[0098] Specifically, with Figure 4Taking the target multi-object image detection model shown as an example, it includes a network backbone and at least two parallel object recognition network branches. The at least two parallel object recognition network branches include a second branch for detecting dirty objects on the ground and obstacles, and a first branch for detecting furniture objects and work scenes. Inputting the image to be detected into the target multi-object image detection model can include: the network backbone of the target multi-object image detection model performing three-layer convolution processing on the image to be detected to obtain a global feature map with a pixel size of 180*180. This feature map is then input into the first branch for furniture object detection and work scene object recognition. The lower half of the feature map is extracted as a local feature map, and this local feature map is input into the second branch for detecting and recognizing dirty objects on the ground and obstacles. The target multi-object image detection model can ultimately output the target objects in the image to be detected, namely, dirty objects on the ground and their positions in the image, obstacles and their positions in the image, furniture objects and their positions in the image, and the work scene represented by the image to be detected. Based on the above information, the mobile robot can adopt the corresponding cleaning mode in the current work scenario to clean the dirt on the ground and can avoid indoor furniture and obstacles on its own.

[0099] According to the image detection method of the embodiments of this specification, an image to be detected is acquired, and the image to be detected is input into the trained target multi-object image detection model for detection to obtain the target objects included in the image to be detected. The embodiments of this specification enable a mobile robot to use the target multi-object image detection model to detect multiple objects in an image, improving the speed of indoor detection by the mobile robot.

[0100] Corresponding to the above embodiments, this specification also provides an electronic device, which in some examples can be a mobile robot. Here, a mobile robot is used as an example for description: Figure 6 As shown, the mobile robot 600 includes: a sensing device 610, a memory 630, a processor 620, and a target detection program 640 of the mobile robot 600 stored in the memory 630. The image detection program 640 of the mobile robot 600 is executed by the processor 620 to implement the steps of the image detection method as described in the above embodiments.

[0101] According to the embodiments of this specification, when the image detection program 640 of the mobile robot 600 is executed by the processor 620, the mobile robot 600 can use the target multi-object image detection model to complete the detection of multiple target objects in the image, thereby improving the indoor detection speed of the mobile robot 600.

[0102] Corresponding to the above embodiments, embodiments of this specification also provide an electronic device, such as... Figure 7As shown, it includes a memory 720, a processor 710, and a model training program 730 stored in the memory 720 and capable of running on the processor 710. When the processor 710 executes the model training program 730, it implements the model training method as described in any of the above embodiments.

[0103] According to the embodiments of this specification, when the processor 710 executes the model training program 730, the mobile robot can use the target multi-object image detection model to complete the detection of multiple target objects in the image, thereby improving the speed of indoor detection by the mobile robot.

[0104] Corresponding to the above embodiments, embodiments of this specification also propose a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the model training method or the image detection method of any of the above embodiments.

[0105] According to embodiments of this specification, a computer-readable storage medium, when a model training program or an image detection program is executed by a processor, enables a mobile robot to use a target multi-task image detection model to detect multiple target objects in an image, thereby improving the indoor detection speed of the sweeping robot.

[0106] It should be illustratively shown that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequential list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0107] It should be understood that various parts of this specification can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0108] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0109] In the description of this specification, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this specification and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this specification.

[0110] Furthermore, the terms "first," "second," etc., used in the embodiments of this specification are for descriptive purposes only and should not be construed as indicating or implying relative importance, or implicitly specifying the number of technical features indicated in this embodiment. Therefore, features defined with terms such as "first" and "second" in the embodiments of this specification can explicitly or implicitly indicate that the embodiment includes at least one of those features. In the description of this specification, the word "multiple" means at least two or more, such as two, three, four, etc., unless otherwise explicitly specified in the embodiments.

[0111] In this specification, unless otherwise explicitly specified or limited in the embodiments, the terms "installation," "connection," "joining," and "fixing," etc., appearing in the embodiments, should be interpreted broadly. For example, a connection can be a fixed connection, a detachable connection, or an integral part; it can also be a mechanical connection, an electrical connection, etc. Of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication between two components, or the interaction between two components. Those skilled in the art will be able to understand the specific meaning of the above terms in this specification based on the specific implementation.

[0112] In this specification, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0113] Although embodiments of this specification have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting this specification. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this specification.

Claims

1. A model training method, characterized in that, The method includes: A training image sample containing at least one object is input into a multi-object image detection model to obtain object information of each object in the training image sample. The multi-object image detection model includes at least two object recognition network branches, which are used to detect object information of objects in the input image. Based on the preset object information labels of objects in the training image samples and the object information of each object in the training image samples, the branch loss of the object recognition network branch is determined; The loss weight coefficient of the object recognition network branch is adjusted according to the source attribute of the preset object information tag; The loss of the multi-object image detection model is determined based on the branch loss of the object recognition network branch and the adjusted loss weight coefficient, so as to satisfy the model training stopping condition and obtain the target multi-object image detection model. The step of adjusting the loss weight coefficient of the object recognition network branch according to the source attribute of the preset object information tag includes: When the source attribute of the preset object information tag is the first preset attribute, increase the loss weight coefficient of the object recognition network branch that detects the object corresponding to the preset object information tag; or, When the source attribute of the preset object information tag is the second preset attribute, the loss weight coefficient of the object recognition network branch that detects the object corresponding to the preset object information tag is reduced; wherein, the confidence level of the first preset attribute is greater than the confidence level of the second preset attribute; The multi-object image detection model includes at least two object recognition network branches from the following: dirt detection network branch, obstacle detection network branch, furniture detection network branch, and work scene classification network branch.

2. The method according to claim 1, characterized in that, The multi-object image detection model includes a first object recognition network branch for detecting a first type of object and a second object recognition network branch for detecting a second type of object; The step of adjusting the loss weight coefficient of the object recognition network branch according to the source attribute of the preset object information tag includes: If the source attribute of the preset object information tag corresponding to the second type of object is the second preset attribute, then based on the source attribute of the preset object information tag corresponding to the second type of object, the loss weight coefficient of the second object recognition network branch is reduced, and / or the loss weight coefficient of the first object recognition network branch is increased. If the source attribute of the preset object information tag corresponding to the first type of object is the first preset attribute, then based on the source attribute of the preset object information tag corresponding to the first type of object, the loss weight coefficient of the second object recognition network branch is reduced, and / or the loss weight coefficient of the first object recognition network branch is increased.

3. The method according to claim 1, characterized in that, The multi-object image detection model includes a task scene classification network branch. Adjusting the loss weight coefficients of the object recognition network branch based on the source attributes of the preset object information labels includes: After the training image samples are input into the job scene classification network branch, if the job scene category output in the training image samples is unclear, then the preset coefficient threshold is set as the loss weight coefficient used for the job scene classification network branch.

4. The method according to any one of claims 1-3, characterized in that, The step of inputting training image samples, including at least one object, into the constructed multi-object image detection model to obtain object information of each object in the training image samples includes: The feature maps of the training image samples are determined by performing convolution processing on the training image samples. The feature map is input into at least two object recognition network branches according to preset rules; The object recognition network branches detect the object information of the objects included in the input feature map.

5. The method according to claim 4, characterized in that, The step of inputting the feature map into the at least two object recognition network branches according to preset rules includes: The feature map is input into one of the object recognition network branches of the multi-object image detection model, and the part of the feature map from which no object is detected is extracted to obtain a local feature map. The local feature map is then input into other object recognition network branches in the multi-object image detection model. Alternatively, at least one local feature map can be obtained by cropping a specified portion from the feature map, and the at least one local feature map and the feature map can be input into each object recognition network branch of the multi-object image detection model according to a preset correspondence.

6. An image detection method, characterized in that, The method includes: Acquire the image to be detected; The image to be detected is input into the target multi-object image detection model obtained by any one of claims 1-5 for detection, and the object information of the target objects included in the image to be detected is obtained.

7. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the model training method as described in any one of claims 1-5 and / or the image detection method as described in claim 6.

8. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the model training method as described in any one of claims 1-5 and / or the image detection method as described in claim 6.

Images