Target positioning marking method and device, storage medium and intelligent device

Abstract

The application is suitable for the technical field of intelligent devices, and provides a target object positioning and marking method and device, a storage medium and an intelligent device. The method comprises the following steps: acquiring an environment image in the visual range of the intelligent device; inputting the environment image into a pre-trained target detection model to obtain a target detection frame and an object category of a target object in the target detection frame; calculating the distance and angle of the target object relative to the intelligent device according to the target detection frame; and positioning and marking the target object based on the object category, the distance and the angle. The application can accurately identify and position and mark objects, and realizes accurate cleaning.

Description

Technical Field

[0001] This application relates to the field of smart device technology, and in particular to a target object positioning and marking method, apparatus, storage medium and smart device. Background Technology

[0002] With the rapid development of technology, various smart home appliances have emerged. Smart home appliances have brought great convenience to people's daily lives, and robot vacuum cleaners are one of them. For robot vacuum cleaners, the home environment is complex and ever-changing. During the cleaning process, robot vacuum cleaners inevitably encounter obstacles, some of which are very valuable or even fragile. If the robot vacuum cleaner does not accurately identify and locate the obstacle, it may collide with it, or mistakenly clean the obstacle as something that needs to be cleaned, easily causing damage to the obstacle.

[0003] Therefore, accurately identifying and locating marked objects to achieve precise cleaning is a problem that needs to be considered. Summary of the Invention

[0004] This application provides a target object positioning and marking method, apparatus, storage medium, and smart device, which can accurately identify and locate marked objects, achieve precise cleaning, and minimize damage to non-target objects.

[0005] In a first aspect, embodiments of this application provide a target object location marking method, including:

[0006] Acquire environmental images within the field of view of the intelligent device;

[0007] The environmental image is input into a pre-trained target detection model to obtain target detection boxes and the object categories of the target objects in the target detection boxes.

[0008] Based on the target detection box, calculate the distance and angle of the target object relative to the smart device;

[0009] The target object is located and marked based on the object category, the distance, and the angle.

[0010] In one possible implementation of the first aspect, calculating the distance of the target object relative to the smart device based on the target detection box includes:

[0011] Obtain the pixel coordinates of the first specified position in the target detection box;

[0012] The distance corresponding to the pixel coordinates of the first specified position is queried in the preset distance query image, wherein the preset distance query image includes the mapping relationship between pixel coordinates and distances in the image captured by the camera of the smart device;

[0013] The distance obtained from the query is determined as the distance between the target object and the smart device.

[0014] In one possible implementation of the first aspect, determining the queried distance as the distance of the target object relative to the smart device includes:

[0015] Obtain the preset pixel coordinates corresponding to the pixel coordinates at the first specified position in the preset area verification image. The preset area verification image is used to verify whether the preset pixel coordinates corresponding to the pixel coordinates are within the preset area.

[0016] If the preset pixel coordinates corresponding to the pixel coordinates of the first specified position are within the preset area, then the distance found is determined as the distance of the target object relative to the smart device;

[0017] If the preset pixel coordinates corresponding to the pixel coordinates of the first specified position are not within the preset area, then the preset distance is determined as the distance between the target object and the smart device.

[0018] In one possible implementation of the first aspect, before querying the distance corresponding to the pixel coordinates of the specified location in the preset distance query image, the method includes:

[0019] The camera of the smart device is calibrated to obtain the camera's intrinsic parameters.

[0020] The camera of the smart device is calibrated to obtain the camera extrinsic parameters of the smart device;

[0021] Based on the camera's intrinsic and extrinsic parameters, a preset distance query image and a preset area verification image are established.

[0022] In one possible implementation of the first aspect, calculating the angle of the target object relative to the smart device based on the target detection box includes:

[0023] Obtain the pixel coordinates of the second specified position of the target detection box;

[0024] The camera of the smart device is calibrated to obtain the camera's intrinsic parameters.

[0025] The angle of the target object relative to the smart device is calculated based on the pixel coordinates of the second specified position and the camera parameters.

[0026] In one possible implementation of the first aspect, before inputting the environment image into a pre-trained object detection model to obtain an object detection box and the object category of the object within the object detection box, the method further includes:

[0027] Obtain the initial detection model;

[0028] The initial detection model is trained based on a preset activation function;

[0029] The initial detection model, after training, is pruned using a preset pruning algorithm to obtain the target detection model.

[0030] In one possible implementation of the first aspect, the object detection model further outputs a confidence score, and the step of inputting the environmental image into a pre-trained object detection model to obtain an object detection box and the object category of the object within the object detection box further includes...

[0031] Target objects with confidence levels below a preset confidence threshold and their target detection frames are filtered out, while target objects with confidence levels greater than or equal to the preset confidence threshold and their target detection frames are retained.

[0032] Secondly, embodiments of this application provide a target object positioning marker device, applied to a smart device, comprising:

[0033] An environmental image acquisition unit is used to acquire environmental images within the field of view of the intelligent device.

[0034] The target detection unit is used to input the environmental image into a pre-trained target detection model to obtain a target detection box and the object category of the target object in the target detection box.

[0035] The distance and angle calculation unit is used to calculate the distance and angle of the target object relative to the smart device based on the target detection box;

[0036] A positioning and marking unit is used to locate and mark the target object based on the object category, the distance, and the angle.

[0037] In one possible implementation of the second aspect, the distance angle calculation unit includes:

[0038] The first pixel coordinate acquisition module is used to acquire the pixel coordinates of the first specified position in the target detection box;

[0039] The distance lookup module is used to query the distance corresponding to the pixel coordinates of the first specified position in a preset distance query image, wherein the preset distance query image includes the mapping relationship between pixel coordinates and distances in the image captured by the camera of the smart device;

[0040] The first distance determination module is used to determine the queried distance as the distance between the target object and the smart device.

[0041] In one possible implementation of the second aspect, the distance angle calculation unit further includes:

[0042] The pixel coordinate acquisition module is used to acquire the preset pixel coordinates corresponding to the pixel coordinates at the first specified position in the preset area verification image, wherein the preset area verification image is used to verify whether the preset pixel coordinates corresponding to the pixel coordinates are within the preset area.

[0043] The first distance determination module is further configured to determine the distance found as the distance between the target object and the smart device if the preset pixel coordinates corresponding to the pixel coordinates of the first specified position are within the preset area;

[0044] The second distance determination module is used to determine the preset distance as the distance between the target object and the smart device if the preset pixel coordinates corresponding to the pixel coordinates of the first specified position are not within the preset area.

[0045] In one possible implementation of the second aspect, the target positioning marker device includes

[0046] The first parameter acquisition unit is used to perform intrinsic parameter calibration on the camera of the smart device to obtain the camera intrinsic parameters of the smart device.

[0047] The second parameter acquisition unit is used to perform external parameter calibration on the camera of the smart device to obtain the camera external parameters of the smart device.

[0048] The preset information establishment unit is used to establish a preset distance query image and a preset area verification image based on the camera's internal parameters and external parameters.

[0049] In one possible implementation of the second aspect, the distance angle calculation unit further includes:

[0050] The second pixel coordinate acquisition module is used to acquire the pixel coordinates of the second specified position of the target detection box;

[0051] An internal parameter acquisition module is used to perform internal parameter calibration on the camera of the smart device to obtain the camera internal parameters of the smart device.

[0052] An angle determination module is used to calculate the angle of the target object relative to the smart device based on the pixel coordinates of the second specified position and the camera's internal parameters.

[0053] In one possible implementation of the second aspect, the target object positioning and marking device further includes an initial model acquisition unit for acquiring an initial detection model;

[0054] The model training unit is used to train the initial detection model based on a preset activation function.

[0055] The target detection model acquisition unit is used to prune the trained initial detection model using a preset pruning algorithm to obtain the target detection model.

[0056] In one possible implementation of the second aspect, the target detection model further outputs a confidence score, and the target detection unit includes:

[0057] The target object filtering unit is used to filter out target objects and their target detection frames with a confidence level lower than a preset confidence threshold, and retain target objects and their target detection frames with a confidence level greater than or equal to the preset confidence threshold.

[0058] Thirdly, embodiments of this application provide an intelligent device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target object positioning and marking method as described in the first aspect above.

[0059] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the target object positioning and marking method as described in the first aspect above.

[0060] Fifthly, embodiments of this application provide a computer program product that, when run on a smart device, causes the smart device to execute the target object positioning and marking method as described in the first aspect above.

[0061] In this embodiment, an environmental image within the field of view of the smart device is acquired, and then the environmental image is input into a pre-trained target detection model to obtain a target detection box and the object category of the target object within the target detection box. Based on the target detection box, the distance and angle of the target object relative to the smart device are calculated. Finally, based on the object category, the distance, and the angle, the target object is located and marked. This solution can accurately identify target objects and accurately locate them based on the identified target detection box, thus enabling precise cleaning, minimizing damage to non-target objects, and improving the effectiveness of the smart device in performing cleaning tasks. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart illustrating the implementation of the target object positioning and marking method provided in the embodiments of this application;

[0064] Figure 2 This is a flowchart illustrating a specific implementation of the target detection model in the target location marking method provided in this application embodiment;

[0065] Figure 3 This is a flowchart illustrating a specific implementation of the distance calculation method in the target object positioning and marking method provided in this application embodiment;

[0066] Figure 4 This is a flowchart illustrating a specific implementation of the target object positioning and marking method provided in this application for determining the distance of a target object relative to a smart device;

[0067] Figure 5 This is a flowchart illustrating a specific implementation of the target object positioning and marking method provided in this application, which establishes a preset distance query image and a preset area verification image.

[0068] Figure 5.1 This is a schematic diagram of a scenario for calibrating a smart device camera in the target object positioning and marking method provided in the embodiments of this application;

[0069] Figure 6 This is a flowchart illustrating a specific implementation of the angle calculation method in the target object positioning and marking method provided in this application embodiment;

[0070] Figure 7 This is a structural block diagram of the target object positioning marking device provided in the embodiments of this application;

[0071] Figure 8 This is a schematic diagram of the smart device provided in the embodiments of this application. Detailed Implementation

[0072] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0073] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0074] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0075] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0076] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0077] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0078] The target object positioning and marking method provided in this application embodiment can be applied to intelligent devices such as robot vacuum cleaners. This application embodiment does not impose any restrictions on the specific type of intelligent device.

[0079] Figure 1 The implementation flow of the target object positioning and marking method provided in this application embodiment is illustrated. The method flow includes steps S101 to S104. The specific implementation principle of each step is as follows:

[0080] Step S101: Obtain an environmental image within the field of view of the smart device.

[0081] In this embodiment, the smart device moves based on the cleaning task and acquires environmental images within its field of view during movement. The size of the field of view is related to the parameters of the smart device's camera.

[0082] In some implementations, raw images captured by a camera on a mobile smart device are acquired, and these raw images are preprocessed to obtain an environmental image containing the target object. The preprocessing includes filtering and noise reduction. By preprocessing the raw images captured by the smart device's camera, interference factors such as noise present in the raw images are eliminated, thereby improving the clarity of the acquired environmental image and enhancing the accuracy of target object detection and recognition.

[0083] In other possible implementations, preprocessing may include performing image enhancement processing on the original image to enhance useful information in the environmental image. The aim is to improve the visual effect of the environmental image, purposefully emphasize the overall or local characteristics of the environmental image, make an originally unclear environmental image clearer, or emphasize certain features of interest (target objects), amplify the differences between features of different objects in the environmental image, and suppress features of little interest, thereby improving image quality and enriching information content. For example, image enhancement algorithms may utilize spatial domain methods and frequency domain methods, as detailed in existing technologies, which will not be elaborated here.

[0084] Step S102: Input the environmental image into the pre-trained target detection model to obtain the target detection box and the object category of the target object in the target detection box.

[0085] The object detection model is a neural network model used to detect objects. It extracts feature vectors from the target image based on preset feature extraction rules. The feature vectors of the environmental image are then input into the pre-trained object detection model to obtain the detection result of the environmental image. The detection result includes the identified target detection boxes and the object categories of the objects within the detection boxes. The target object is the item to be cleaned in the cleaning task, and the object category is the category of the item to be cleaned.

[0086] In this embodiment of the application, feature vectors of target objects are extracted from the environmental image according to preset feature extraction rules, and then the feature vectors are input into the trained target detection model to obtain target detection boxes and object categories of target objects in the target detection boxes.

[0087] In some implementations, the environmental image includes one or more target objects.

[0088] As one embodiment of this application, Figure 2 The specific implementation flow of obtaining the target detection model in the target object localization and marking method provided in the embodiments of this application is illustrated below:

[0089] A1: Obtain the initial detection model. The initial detection model can be an existing object detection network model. For example, the tiny model in the YOLOv algorithm can be selected as the initial detection model.

[0090] A2: Based on the preset activation function, train the initial detection model.

[0091] In this embodiment, the original activation function in the initial detection model is replaced, the complex activation function is modified into a simple activation function, and then the initial detection model with the modified activation function is used for training. The training process can refer to the training process of a general neural network model, which will not be described in detail here.

[0092] For example, the preset activation function can be ReLU or LReLU. After modifying the high activation function in the initial detection model to ReLU or LReLU, the initial detection model is trained.

[0093] A3: Using a preset pruning algorithm, the initial detection model that has been trained is pruned to obtain the target detection model.

[0094] In this embodiment, a preset pruning algorithm is used to remove channels with low importance from the trained initial detection model. For example, the L1filter pruning algorithm from the NNI library can be used as the preset pruning algorithm. The sparsity parameter is set to 0.5, or the parameter can be adjusted according to the actual effect.

[0095] In one possible implementation, the accuracy of the initial detection model after pruning can be further improved by using a distillation algorithm to address the loss of model accuracy caused by pruning.

[0096] In this embodiment, the detection speed of the model is improved by modifying the activation function, and the model is pruned to simplify the model and improve the inference speed, thereby improving the efficiency of target object detection.

[0097] In this embodiment, the target detection model also outputs a confidence score. The step of inputting the environmental image into the pre-trained target detection model to obtain target detection boxes and the object categories of the targets within those boxes further includes: filtering target objects and their detection boxes with confidence scores below a preset confidence threshold, and retaining target objects and their detection boxes with confidence scores greater than or equal to the preset confidence threshold. Using confidence scores to filter target detection boxes in the environmental image, retaining those with confidence scores that meet the requirements, can improve the effectiveness and efficiency of target object detection.

[0098] Step S103: Calculate the distance and angle of the target object relative to the smart device based on the target detection box.

[0099] In this embodiment, the distance and angle of the target object relative to the smart device are calculated based on the target detection box.

[0100] When the same environmental image contains more than one target detection box (target object), calculate the distance and angle of each target object relative to the smart device.

[0101] As one possible implementation of this application Figure 3 This paper illustrates a specific implementation flow of the target object positioning and marking method provided in this application, which calculates the distance of the target object relative to the smart device based on the target detection box. The details are as follows:

[0102] B1: Obtain the pixel coordinates of the first specified position in the target detection box.

[0103] The first specified position can be the center position of the bottom edge of the target detection box. Specifically, the pixel coordinates of the top left corner and the bottom right corner of the target detection box are obtained, and the pixel coordinates of the center position of the bottom edge of the target detection box can be determined based on the pixel coordinates of the top left corner and the bottom right corner of the target detection box.

[0104] For example, the coordinates of the top left corner of the target detection box (x min ,y min ), lower right corner coordinates (x max ,y max If the coordinates of the midpoint P of the bottom edge of the target detection box are given, then... d (P dx ,P dy In ), where P dx =(x max -x min ) / 2,P dy =y max .

[0105] B2: Query the distance corresponding to the pixel coordinates of the first specified position in the preset distance query image, wherein the preset distance query image includes the mapping relationship between pixel coordinates and distances in the image captured by the camera of the smart device.

[0106] B3: Determine the distance obtained from the query as the distance between the target object and the smart device.

[0107] In this embodiment of the application, the distance corresponding to the first specified position is directly queried in the preset distance query image to quickly determine the distance of the target object relative to the smart device.

[0108] As one embodiment of this application, such as Figure 4 As shown, the step of determining the distance retrieved as the distance between the target object and the smart device includes:

[0109] C1: Obtain the preset pixel coordinates corresponding to the pixel coordinates at the first specified position in the preset area verification image. The preset area verification image is used to verify whether the preset pixel coordinates corresponding to the pixel coordinates are within the preset area.

[0110] C2: If the preset pixel coordinates corresponding to the pixel coordinates of the first specified position are within the preset area, then the found distance is determined as the distance of the target object relative to the smart device. The size of the preset area is determined based on the construction of the preset distance query image and the preset area verification image.

[0111] C3: If the preset pixel coordinates corresponding to the pixel coordinates of the first specified position are not within the preset area, then the preset distance is determined as the distance between the target object and the smart device. The preset distance is generally a relatively large value, for example, 10m.

[0112] In this embodiment, while querying the distance corresponding to the pixel coordinates of the first specified position in the preset distance query image, it is also necessary to verify whether the pixel coordinates corresponding to the pixel coordinates of the first specified position on the target detection box are within a preset area. If the corresponding pixel coordinates are not within the preset area, the distance between the target object in the target detection box and the smart device is not calculated, but the preset distance is directly assigned.

[0113] For example, the distance corresponding to the midpoint coordinate Pd of the bottom edge of the target detection box is queried in the preset distance query image. The pixel coordinates of Pd in ​​the preset area verification image are obtained. If the pixel coordinates are (255, 255, 255), the distance queried in the preset distance query image is directly determined as the distance of the target object in the target detection box relative to the smart device. If the pixel coordinates are (0, 0, 0), it means that the target object is not in the valid area, and the default distance of the target object in the target detection box relative to the smart device is directly set to a larger value, such as 10m.

[0114] As one embodiment of this application, Figure 5 The following is a detailed implementation flow of the target object positioning and marking method provided in this application, which involves establishing a preset distance query image and a preset region verification image:

[0115] D1: Perform intrinsic parameter calibration on the camera of the smart device to obtain the camera's intrinsic parameters. Intrinsic parameters (or simply intrinsic parameters) are parameters related to the camera's own characteristics, such as focal length and pixel size.

[0116] For example, such as Figure 5.1 As shown, the checkerboard calibration method can be used to calibrate the intrinsic parameters of the camera of the smart device, thereby obtaining the camera's intrinsic parameters. The smart device with the camera to be calibrated is placed on a level surface, and the calibration plate with Aruco codes is placed horizontally in front of the camera. The Aruco codes are approximately 10-30cm away from the camera. The size of the Aruco codes depends on the camera's field of view and the distance between the calibration plate and the camera, ensuring that the camera can capture the complete pattern of the Aruco codes without making them too small. In this application, the actual size used can be 14cm*14cm.

[0117] D2: Perform extrinsic parameter calibration on the camera of the smart device to obtain the camera's extrinsic parameters. Camera extrinsic parameters (or extrinsic parameters for short) are parameters in the world coordinate system, such as the camera's position and rotation direction. Unlike the constant intrinsic parameters, extrinsic parameters change with camera movement.

[0118] For example, the camera extrinsic parameters are calibrated using the Aruco library to obtain the rotation matrix R and translation matrix T, with the x-axis and y-axis as shown below. Figure 5.1 As shown, the positive z-axis is perpendicular to the ground and points upwards, with the origin of the world coordinate system being the center of the Aruco code.

[0119] D3: Based on the camera's intrinsic parameters and extrinsic parameters, establish a preset distance query image and a preset area verification image.

[0120] In this embodiment, after obtaining the camera's intrinsic and extrinsic parameters, points in the world coordinate system can be transformed to the camera coordinate system based on the world coordinate to camera coordinate transformation relationship. Since the camera's intrinsic parameters are also known, camera coordinates can be converted to pixel coordinates. Therefore, for a point on the ground, the center of the calibration board is the origin of the world coordinate system, with coordinates (0, 0, 0). A world coordinate range is set, such as X... left =-1,X right =1,Y up =1,Y down = -0.3 (all units are meters), which can be used to define a region on the ground, namely the preset region mentioned above. The coordinates of the four vertices of the region are (-1, 1, 0), (1, 1, 0), (-1, -0.3, 0), and (1, -0.3, 0). The z-value of all points in this region is 0, and only the x and y values ​​change. By traversing all points in this region (the step size can be set to 1 millimeter), the camera coordinates and pixel coordinates corresponding to each point are calculated, and the calculated distance value is written into the data to obtain a preset distance query image.

[0121] In this embodiment, the preset area verification image is mainly used to define the area range. The pixel coordinates within the preset area are (255, 255, 255), and the pixel coordinates outside the preset area are (0, 0, 0).

[0122] In this embodiment of the application, a single-channel preset distance query image I is initialized. distance And three-channel preset area verification image I mask All values ​​are initialized to 0, and the resolution is the same as the resolution of the image captured by the camera (e.g., 640, 480). For world coordinates (-1, 1, 0), the camera coordinates (X...) are calculated. c ,Y c Z c ) and pixel coordinates (U,V), I distance The value at position (U,V) is updated from 0 to Z. c Z c I represents the distance corresponding to the pixel coordinates. mask Fill the (U,V) position with a white circle of 5 pixels radius, updating the value from (0, 0, 0) to (255, 255, 255). Then move 1mm to the right along the X-axis to obtain the world coordinates (-0.999, 1, 0). Similarly, calculate the corresponding camera coordinates and pixel coordinates, and update I. distance and I mask The preset distance query image and preset area verification image will be obtained after traversing all world coordinates within the preset area.

[0123] In one possible implementation, preset distance query images and preset region verification images are established for different object categories. That is, the preset distance query images and preset region verification images are different for target objects of different categories. This can further improve the accuracy of distance calculation.

[0124] As one possible implementation of this application Figure 6 This paper illustrates a specific implementation flow of the target object positioning and marking method provided in this application, which calculates the angle of the target object relative to the smart device based on the target detection box. The details are as follows:

[0125] E1: Obtain the camera coordinates of the second specified position of the target detection box.

[0126] The second specified position can be the center point of the target detection box. Specifically, the pixel coordinates of the top left corner and the bottom right corner of the target detection box are obtained, and the pixel coordinates of the center position of the bottom edge of the target detection box can be determined based on the pixel coordinates of the top left corner and the bottom right corner of the target detection box.

[0127] For example, the coordinates of the top left corner of the target detection box (xmin ,y min ), lower right corner coordinates (x max ,y max If the coordinates of the midpoint P of the bottom edge of the target detection box are given, then... c (P cx ,P cy In ), where P cx =(x max -x min ) / 2,P cy =(y max -y min ) / 2.

[0128] E2: Perform intrinsic parameter calibration on the camera of the smart device to obtain the camera's intrinsic parameters. The method of intrinsic parameter calibration is the same as described above and will not be repeated here.

[0129] E3: Calculate the angle of the target object relative to the smart device based on the camera coordinates at the second specified location and the camera's intrinsic parameters.

[0130] In this embodiment of the application, the angle of the target object relative to the smart device is calculated based on the camera coordinates of the second specified position of the target detection box and the camera intra-parameters. Specifically, the calculation can be performed with reference to existing technologies.

[0131] Step S104: Based on the object category, the distance, and the angle, locate and mark the target object.

[0132] In this embodiment, the target object is located and marked based on the object category of the identified target object and the calculated distance and angle of the target object relative to the smart device, so that the smart device can plan a path and improve the accuracy of the cleaning task.

[0133] In some implementations, if multiple targets exist in the same environmental image, each target is located and marked separately. A path is then planned based on the location marks of each target.

[0134] In this embodiment, an environmental image within the field of view of the smart device is acquired, and then the environmental image is input into a pre-trained target detection model to obtain a target detection box and the object category of the target object within the target detection box. Based on the target detection box, the distance and angle of the target object relative to the smart device are calculated. Finally, based on the object category, the distance, and the angle, the target object is located and marked. This solution can accurately identify target objects and accurately locate them based on the identified target detection box, thus enabling precise cleaning, minimizing damage to non-target objects, and improving the effectiveness of the smart device in performing cleaning tasks.

[0135] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0136] Corresponding to the target object positioning and marking method described in the above embodiments, Figure 7 A structural block diagram of the target object positioning marking device provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0137] Reference Figure 7 The target positioning and marking device is applied to a smart device and includes: an environmental image acquisition unit 71, a target detection unit 72, a distance and angle calculation unit 73, and a positioning and marking unit 74, wherein:

[0138] The environmental image acquisition unit 71 is used to acquire environmental images within the field of view of the smart device.

[0139] The target detection unit 72 is used to input the environmental image into a pre-trained target detection model to obtain a target detection box and the object category of the target object in the target detection box;

[0140] The distance and angle calculation unit 73 is used to calculate the distance and angle of the target object relative to the smart device based on the target detection box;

[0141] The positioning marking unit 74 is used to locate and mark the target object based on the object category, the distance, and the angle.

[0142] As one possible implementation of this application, the distance angle calculation unit 73 includes:

[0143] The first pixel coordinate acquisition module is used to acquire the pixel coordinates of the first specified position in the target detection box;

[0144] The distance lookup module is used to query the distance corresponding to the pixel coordinates of the first specified position in a preset distance query image, wherein the preset distance query image includes the mapping relationship between pixel coordinates and distances in the image captured by the camera of the smart device;

[0145] The first distance determination module is used to determine the queried distance as the distance between the target object and the smart device.

[0146] As one possible implementation of this application, the distance angle calculation unit 73 further includes:

[0147] The pixel coordinate acquisition module is used to acquire the preset pixel coordinates corresponding to the pixel coordinates at the first specified position in the preset area verification image, wherein the preset area verification image is used to verify whether the preset pixel coordinates corresponding to the pixel coordinates are within the preset area.

[0148] The first distance determination module is further configured to determine the distance found as the distance between the target object and the smart device if the preset pixel coordinates corresponding to the pixel coordinates of the first specified position are within the preset area;

[0149] The second distance determination module is used to determine the preset distance as the distance between the target object and the smart device if the preset pixel coordinates corresponding to the pixel coordinates of the first specified position are not within the preset area.

[0150] As one possible implementation of this application, the above-mentioned target positioning marker device includes

[0151] The first parameter acquisition unit is used to perform intrinsic parameter calibration on the camera of the smart device to obtain the camera intrinsic parameters of the smart device.

[0152] The second parameter acquisition unit is used to perform external parameter calibration on the camera of the smart device to obtain the camera external parameters of the smart device.

[0153] The preset information establishment unit is used to establish a preset distance query image and a preset area verification image based on the camera's internal parameters and external parameters.

[0154] As one possible implementation of this application, the distance angle calculation unit 73 further includes:

[0155] The second pixel coordinate acquisition module is used to acquire the pixel coordinates of the second specified position of the target detection box;

[0156] An internal parameter acquisition module is used to perform internal parameter calibration on the camera of the smart device to obtain the camera internal parameters of the smart device.

[0157] An angle determination module is used to calculate the angle of the target object relative to the smart device based on the pixel coordinates of the second specified position and the camera's internal parameters.

[0158] As one possible implementation of this application, the above-mentioned target positioning marker device further includes

[0159] The initial model acquisition unit is used to acquire the initial detection model;

[0160] The model training unit is used to train the initial detection model based on a preset activation function.

[0161] The target detection model acquisition unit is used to prune the trained initial detection model using a preset pruning algorithm to obtain the target detection model.

[0162] As one possible implementation of this application, the target detection model also outputs a confidence score, and the target detection unit 72 includes:

[0163] The target object filtering unit is used to filter out target objects and their target detection frames with a confidence level lower than a preset confidence threshold, and retain target objects and their target detection frames with a confidence level greater than or equal to the preset confidence threshold.

[0164] In this embodiment, an environmental image within the field of view of the smart device is acquired, and then the environmental image is input into a pre-trained target detection model to obtain a target detection box and the object category of the target object within the target detection box. Based on the target detection box, the distance and angle of the target object relative to the smart device are calculated. Finally, based on the object category, the distance, and the angle, the target object is located and marked. This solution can accurately identify target objects and accurately locate them based on the identified target detection box, thus enabling precise cleaning, minimizing damage to non-target objects, and improving the effectiveness of the smart device in performing cleaning tasks.

[0165] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0166] This application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements... Figures 1 to 6 The steps of any target object location marking method are represented.

[0167] This application embodiment also provides a smart device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements... Figures 1 to 6 The steps of any target object location marking method are represented.

[0168] This application also provides a computer program product that, when run on a server, causes the server to execute the implementation as described above. Figures 1 to 6 The steps of any target object location marking method are represented.

[0169] Figure 8 This is a schematic diagram of a smart device provided in an embodiment of this application. Figure 8As shown, the intelligent device 8 in this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. When the processor 80 executes the computer program 82, it implements the steps in the various target object positioning and marking method embodiments described above, for example... Figure 1 Steps S101 to S104 are shown. Alternatively, when the processor 80 executes the computer program 82, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 7 The functions of units 71 to 74 shown.

[0170] For example, the computer program 82 may be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program 82 in the smart device 8.

[0171] The intelligent device 8 can be a smart robotic vacuum cleaner. The intelligent device 8 may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that... Figure 8 This is merely an example of the smart device 8 and does not constitute a limitation on the smart device 8. It may include more or fewer components than shown, or combine certain components, or different components. For example, the smart device 8 may also include input / output devices, network access devices, buses, etc.

[0172] The processor 80 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0173] The memory 81 can be an internal storage unit of the smart device 8, such as a hard drive or memory of the smart device 8. The memory 81 can also be an external storage device of the smart device 8, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the smart device 8. Furthermore, the memory 81 can include both internal and external storage units of the smart device 8. The memory 81 is used to store the computer program and other programs and data required by the smart device. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0174] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0175] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0176] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0177] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0178] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for locating and marking a target object, characterized in that, Applied to smart devices, including: Acquire environmental images within the field of view of the intelligent device; The environmental image is input into a pre-trained target detection model to obtain target detection boxes and the object categories of the target objects in the target detection boxes. Calculating the distance and angle of the target object relative to the smart device based on the target detection box includes: obtaining the pixel coordinates of a first specified position in the target detection box; performing intrinsic parameter calibration on the camera of the smart device to obtain the camera intrinsic parameters of the smart device; performing extrinsic parameter calibration on the camera of the smart device to obtain the camera extrinsic parameters of the smart device; establishing a preset distance query image and a preset region verification image based on the camera intrinsic parameters and the camera extrinsic parameters; querying the distance corresponding to the pixel coordinates of the first specified position in the preset distance query image, wherein the preset distance query image includes the mapping relationship between pixel coordinates and distances in the image captured by the camera of the smart device; and determining the queried distance as the distance of the target object relative to the smart device. The target object is located and marked based on the object category, the distance, and the angle.

2. The method according to claim 1, characterized in that, Determining the queried distance as the distance between the target object and the smart device includes: Obtain the preset pixel coordinates corresponding to the pixel coordinates at the first specified position in the preset area verification image. The preset area verification image is used to verify whether the preset pixel coordinates corresponding to the pixel coordinates are within the preset area. If the preset pixel coordinates corresponding to the pixel coordinates of the first specified position are within the preset area, then the distance found is determined as the distance of the target object relative to the smart device; If the preset pixel coordinates corresponding to the pixel coordinates of the first specified position are not within the preset area, then the preset distance is determined as the distance between the target object and the smart device.

3. The method according to claim 1, characterized in that, The step of calculating the angle of the target object relative to the smart device based on the target detection box includes: Obtain the pixel coordinates of the second specified position of the target detection box; The camera of the smart device is calibrated to obtain the camera's intrinsic parameters. The angle of the target object relative to the smart device is calculated based on the pixel coordinates of the second specified position and the camera parameters.

4. The method according to claim 1, characterized in that, Before inputting the environmental image into a pre-trained object detection model to obtain object detection boxes and object categories of objects within those boxes, the method further includes: Obtain the initial detection model; The initial detection model is trained based on a preset activation function; The initial detection model, after training, is pruned using a preset pruning algorithm to obtain the target detection model.

5. The method according to any one of claims 1 to 4, characterized in that, The target detection model also outputs a confidence score. The process of inputting the environmental image into the pre-trained target detection model to obtain a target detection box and the object category of the target object within the detection box also includes... Target objects with confidence levels below a preset confidence threshold and their target detection frames are filtered out, while target objects with confidence levels greater than or equal to the preset confidence threshold and their target detection frames are retained.

6. A target object positioning and marking device, characterized in that, Applied to smart devices, including: An environmental image acquisition unit is used to acquire environmental images within the field of view of the intelligent device. The target detection unit is used to input the environmental image into a pre-trained target detection model to obtain a target detection box and the object category of the target object in the target detection box. The distance and angle calculation unit is used to calculate the distance and angle of the target object relative to the smart device based on the target detection box; The first parameter acquisition unit is used to perform intrinsic parameter calibration on the camera of the smart device to obtain the camera intrinsic parameters of the smart device. The second parameter acquisition unit is used to perform external parameter calibration on the camera of the smart device to obtain the camera external parameters of the smart device. The preset information establishment unit is used to establish a preset distance query image and a preset area verification image based on the camera's internal parameters and external parameters. The distance angle calculation unit includes: The first pixel coordinate acquisition module is used to acquire the pixel coordinates of the first specified position in the target detection box; The distance lookup module is used to query the distance corresponding to the pixel coordinates of the first specified position in the preset distance query image, wherein the preset distance query image includes the mapping relationship between pixel coordinates and distances in the image captured by the camera of the smart device; The first distance determination module is used to determine the queried distance as the distance between the target object and the smart device; A positioning and marking unit is used to locate and mark the target object based on the object category, the distance, and the angle.

7. A smart device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the target location marking method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the target location marking method as described in any one of claims 1 to 5.

Images