Object detection methods, apparatus, integrated circuit chips, computing devices, and board cards

Abstract

The present disclosure relates to an object detection method, device, integrated circuit chip, computing device and board card, wherein the computing device can be included in a combined processing device, which can further include a general-purpose interconnection interface and other processing devices. The computing device interacts with the other processing devices to jointly complete a user-specified computing operation. The combined processing device can further include a storage device connected to the computing device and the other processing devices respectively for data of the computing device and the other processing devices. The scheme of the present disclosure can be widely applied to object detection.

Description

Technical Field

[0001] This disclosure generally relates to the field of image detection. More specifically, this disclosure relates to object detection methods, apparatus, integrated circuit chips, computing devices, and circuit boards. Background Technology

[0002] Object detection has wide applications in fields such as intelligent video and autonomous driving. However, current methods for selecting detection models for object recognition often suffer from imbalanced samples, conflicting feature spaces, and neglect of object correlations. This leads to the selection of expert models for different objects to improve detection accuracy, resulting in high computational costs and hindering the implementation of algorithms at edge computing environments. Furthermore, ignoring object correlations results in numerous false positives. Therefore, developing a detection method that addresses feature space conflicts and imbalanced samples while considering object correlations is a key challenge in current technologies. Summary of the Invention

[0003] In order to at least partially solve the technical problems mentioned in the background art, the present disclosure provides an object detection method, an object detection device, an integrated circuit chip including the object detection device, a computing device, and a board.

[0004] In one aspect, this disclosure provides an object detection method, the object detection method comprising: acquiring an image to be detected; obtaining object features associated with at least one object based on the image to be detected; and determining object detection results with local object information for each of the at least one object based on the object features.

[0005] In another aspect, this disclosure provides an object detection apparatus, the object detection apparatus comprising: an acquisition module for acquiring an image to be detected; a feature extraction module for obtaining object features related to at least one object based on the image to be detected; and a detection module for determining an object detection result with local object information for the at least one object based on the object features.

[0006] In another aspect, this disclosure provides an integrated circuit chip that includes the object detection device described above. In yet another embodiment, the object detection device of this disclosure can be configured as a standalone integrated circuit chip.

[0007] In another aspect, this disclosure provides a computing device that includes the object detection device or the integrated circuit chip described above.

[0008] In another aspect, this disclosure provides a board that includes the aforementioned computing device.

[0009] By utilizing the object detection method, object detection device, integrated circuit chip, computing device, and board disclosed herein, it is possible to detect related objects by sharing shallow features and correlation, thereby achieving or even surpassing the detection accuracy using multiple expert models. Therefore, the object detection method of this disclosure significantly reduces the computational power requirement and is more suitable for deployment on edge chips. Attached Figure Description

[0010] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0011] Figure 1 This is a flowchart illustrating an object detection method according to an embodiment of the present disclosure;

[0012] Figure 2 This is a flowchart illustrating an object detection method according to another embodiment of the present disclosure;

[0013] Figure 3 This is a schematic structural framework diagram illustrating a neural network model according to an embodiment of the present disclosure;

[0014] Figure 4 This is a schematic block diagram illustrating an object detection apparatus according to an embodiment of the present disclosure;

[0015] Figure 5 This is a schematic block diagram illustrating an object detection apparatus according to another embodiment of the present disclosure;

[0016] Figure 6 This is a structural diagram illustrating a combined processing apparatus according to an embodiment of the present disclosure; and

[0017] Figure 7 This is a schematic diagram illustrating the structure of a circuit board according to an embodiment of the present disclosure. Detailed Implementation

[0018] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0019] The technical solution disclosed herein provides an object detection method, an object detection device, an integrated circuit chip, a computing device, and a board. Unlike existing object detection methods, this disclosure provides an object detection method that overcomes feature space conflicts and sample imbalance by sharing shallow features and extracting deep correlation features separately. In particular, this disclosure can achieve or even surpass the detection accuracy using multiple expert models. Therefore, the object detection method of this disclosure significantly reduces the computational power requirements and is more suitable for deployment on edge chips.

[0020] The specific embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0021] This disclosure provides an object detection method. (Refer to...) Figure 1 , Figure 1 This is a flowchart illustrating an object detection method according to an embodiment of the present disclosure. Figure 1 As shown, the object detection method includes the following steps S101-S103. Step S101: Acquire an image to be detected. Step S102: Obtain object features associated with at least one object based on the image to be detected. Step S103: Based on the object features, determine object detection results with local object information for each of the at least one object.

[0022] In step S101, the image to be detected can be acquired.

[0023] According to embodiments of this disclosure, the image to be detected is an image used in object detection or recognition, and may include a photograph, picture, or a frame of a video. Furthermore, the image to be detected may be obtained directly using an imaging device, or it may be a pre-stored image to be detected acquired from other components via transmission.

[0024] In step S102, object features associated with at least one object can be obtained from the image to be detected.

[0025] According to embodiments of this disclosure, when features related to one or more objects to be detected exist in the image to be detected, all relevant features of the objects involved can be obtained from the image to be detected. The objects to be detected may include people, various animals such as cats and dogs, vehicles, buildings, etc. Furthermore, multiple objects to be detected may be multiple identical objects, such as multiple people. Here, taking two objects to be detected—a person and a car—as an example, in this step, if features related to a person and / or a car exist in the image to be detected, all relevant features of the person and / or the car will be obtained.

[0026] Furthermore, obtaining object features related to the at least one object from the image to be detected may include: extracting shallow features related to the at least one object from the image to be detected through a backbone network; and fusing the shallow features through a feature fusion network to obtain the object features. This feature fusion network may include various known applicable networks. Therefore, more specifically, multiple layers of shallow features at different scales related to the at least one object can be extracted from the image to be detected through a backbone network, and the multiple layers of shallow features can be fused separately through a feature pyramid network to obtain multi-layer object features.

[0027] In this embodiment, when the image to be detected contains features related to one or more objects to be detected, the backbone network can first extract all relevant shallow features of the objects involved at different scales (dimensions) from the image to be detected. Each scale corresponds to one layer of features. Then, the feature pyramid network can fuse the multiple layers of shallow features to obtain fused multi-layer object features. The backbone network can be a residual network or any other feature extraction network suitable for this purpose, without limitation.

[0028] Taking people and cars as two objects to be detected as an example again. If there are features related to people and cars in the image to be detected, such as the backbone network of the residual network, the backbone network can first extract all relevant shallow features of people and cars at different scales (dimensions) from the image to be detected. Each scale corresponds to one layer of features. Then the feature pyramid network can fuse the multiple layers of shallow features to obtain the fused multi-layer features of people and cars.

[0029] In step S103, object detection results with local object information can be determined for each of the at least one object based on the object features.

[0030] According to embodiments of this disclosure, if only one object to be detected is involved, the detection result is determined based on all relevant features of that single object, i.e., the object and its location (coordinates) are determined. The detection result includes the desired local information of the object, i.e., a local detection result. This local detection result includes the local classification and location (coordinates), and the local classification has been filtered, i.e., it has been determined to be local. If multiple objects to be detected are involved, the detection result for each object is determined based on all relevant features of the multiple objects, i.e., the object and its location (coordinates) are determined. The detection result for each object includes the desired local information of the object, i.e., a local detection result. This local detection result includes the local classification and location (coordinates), and the local classification has been filtered, i.e., it has been determined to be local.

[0031] Taking people and vehicles as examples again, with the human's feature represented by the head and the vehicle's feature represented by the license plate, the detection results are determined based on the obtained features of both the person and the vehicle. The human detection result includes the person and their coordinates, and the vehicle detection result includes the vehicle and its coordinates. Furthermore, the human detection result also includes the head and its coordinates, and the vehicle detection result includes the license plate and its coordinates. For simplicity, using only the human as an example, the human detection result can include 10 parameters: 4 parameters directly or indirectly representing the person's coordinates, 1 parameter representing the person's classification (binary classification, indicating whether it is a person), 4 parameters directly or indirectly representing the head's coordinates, and 1 parameter representing the head's classification (binary classification, indicating whether it is a head). Of course, for the final human detection result, the value of the parameter representing the person's classification represents a person, and the value of the parameter representing the head's classification represents a head.

[0032] Step S103 will be described in more detail below. Specifically, determining object detection results with local object information for each of the at least one object based on the object features may include: obtaining preliminary object detection results with local object information for each of the at least one object through their respective classification and regression networks based on the object features; obtaining intermediate object detection results through non-maximum suppression processing based on the preliminary object detection results; and determining object detection results with local object information based on the intermediate object detection results.

[0033] In this embodiment, if only one object to be detected is involved, preliminary object detection results can be obtained for each layer of object features through a classification and regression network based on the obtained multi-layer object features. In fact, the classification and regression network obtains multiple preliminary object detection results based on each layer of object features. Each preliminary object detection result includes the object's classification (i.e., the object itself) and its location coordinates, and also includes the desired local information about the object. This local information includes information related to the local classification and location coordinates; however, the local classification is not yet filtered in this step, meaning it is not yet determined to be local. To obtain more accurate object detection results, non-maximum suppression processing can be applied to all preliminary object detection results to obtain intermediate object detection results. These intermediate results can be understood as the more accurate preliminary object detection results remaining after processing. Finally, the object detection result is determined from these intermediate results. This object detection result includes local detection results, which include local classification and location coordinates. The local classification has been filtered, meaning it has been determined to be local.

[0034] For cases involving multiple objects to be detected, based on the obtained multi-layer object features of the involved objects (i.e., each layer of object features involves multiple objects), for each layer of object features, each object obtains its preliminary object detection result through its own classification and regression network. In fact, based on each layer of object features, each object obtains multiple preliminary object detection results through its own classification and regression network. Each preliminary object detection result includes the object's classification (i.e., it is the object itself) and the object's location coordinates, and each preliminary object detection result includes the local information of the object to be obtained. This local information includes the local classification and location coordinates; however, the local classification is not yet filtered in this step, meaning it is not yet determined to be local. To obtain more accurate object detection results, non-maximum suppression processing can be applied to all preliminary object detection results of the involved objects to obtain intermediate object detection results for the object. These intermediate object detection results can be understood as the more accurate preliminary object detection results remaining after processing. Finally, the object detection result is determined based on the intermediate object detection results of the involved objects. This object detection result includes local detection results, which include the local classification and location coordinates; the local classification has been filtered, meaning it has been determined to be local.

[0035] Taking people and vehicles as examples again, with the human head as the local feature and the license plate as the local feature of the vehicle, we can first obtain preliminary detection results for people and vehicles by using a classification and regression network for each layer of human and vehicle features. In fact, based on each layer of human and vehicle features, the classification and regression network for people will obtain multiple preliminary detection results, and the classification and regression network for vehicles will obtain multiple preliminary detection results.

[0036] Each preliminary person detection result includes the person's classification (i.e., it is a person) and the person's location coordinates, and each preliminary person detection result includes the desired head information. This head information includes the head's classification and location coordinates; however, the classification of the head is not yet filtered at this step, meaning it is not yet confirmed to be a head. To obtain more accurate person detection results, non-maximum suppression processing can be applied to all preliminary person detection results to obtain intermediate person detection results. These intermediate detection results can be understood as the more accurate preliminary person detection results remaining after processing. Finally, the person detection result is determined by the intermediate person detection results. The person detection result includes a head detection result, which includes the head's classification and location coordinates; the head's classification has been filtered, meaning it has been confirmed to be a head.

[0037] Furthermore, each preliminary vehicle detection result includes the vehicle's classification (i.e., it is a vehicle) and its location coordinates, and also includes the desired license plate information. This license plate information includes the license plate's classification and location coordinates; however, the license plate classification is not yet filtered at this step, meaning it is not yet confirmed as a license plate. To obtain more accurate vehicle detection results, non-maximum suppression processing can be applied to all preliminary vehicle detection results to obtain intermediate vehicle detection results. These intermediate results can be understood as the more accurate preliminary vehicle detection results remaining after processing. Finally, the vehicle detection result is determined by the intermediate vehicle detection results. The vehicle detection result includes the license plate detection result, which includes the license plate's classification and location coordinates. The license plate's classification has been filtered, meaning it has been confirmed as a license plate.

[0038] Furthermore, determining the object detection result containing the local information of the object based on the intermediate object detection result includes: when the product of the classification parameter of the object and the classification parameter of the object in the intermediate object detection result is greater than a preset threshold, the intermediate object detection result is taken as the object detection result containing the local information of the object.

[0039] In this embodiment, in order to filter the local object classification in the intermediate object detection result, the object classification parameter in the intermediate object detection result can be multiplied with the local object classification parameter. If the product is greater than a preset threshold, it indicates that the local object classification is determined to be that local. Therefore, the intermediate object detection result can be used as the object detection result with local object information.

[0040] Taking people and vehicles as examples, with the person's head as the local classification parameter and the vehicle's license plate as the local classification parameter, the classification parameters for the person and the head in the intermediate detection results can be multiplied. If the product is greater than a preset threshold, it indicates that the person is definitely classified as a head, and therefore the intermediate detection result for that person can be used as the detection result for a person containing head information. Similarly, the classification parameters for the vehicle and the license plate in the intermediate detection results can be multiplied. If the product is greater than a preset threshold, it indicates that the license plate is definitely classified as a license plate, and therefore the intermediate detection result for that vehicle can be used as the detection result for a vehicle containing license plate information.

[0041] According to embodiments of this disclosure, taking both people and vehicles as examples of objects to be detected, with the human's portion represented by a head and the vehicle's portion represented by a license plate, all intermediate detection results for people can be stored as a set A, and all intermediate detection results for vehicles can be stored as a set B. The detection result for a person is determined based on set A, and all detection results for a person are stored as set C. The detection result for a vehicle is determined based on set B, and all detection results for a vehicle are stored as set D. Sets A and B represent confirmed detection results for both people and vehicles, but without confirmation regarding whether they contain a head or a license plate. Sets C and D represent confirmed detection results for both people and vehicles, containing both a head and a license plate. Therefore, sets A, B, C, and D can be output as needed.

[0042] This disclosure also provides another object detection method. (See reference...) Figure 2 , Figure 2 This is a flowchart illustrating an object detection method according to another embodiment of the present disclosure. Figure 2 As shown, the object detection method includes the following steps S201-S207.

[0043] Step S201: Train the neural network model consisting of the residual network, the feature pyramid network, and the classification and regression network.

[0044] Step S202: Obtain the image to be detected.

[0045] Step S203: Extract multi-layer shallow features of different scales related to the at least one object from the image to be detected using a residual network.

[0046] Step S204: The multi-layer shallow features are fused separately using a feature pyramid network to obtain multi-layer object features.

[0047] Step S205: Based on the object features, obtain preliminary object detection results with local information of the object for the at least one object through their respective classification and regression networks.

[0048] Step S206: Based on the preliminary object detection results, obtain intermediate object detection results through non-maximum suppression processing.

[0049] Step S207: Based on the intermediate object detection result, determine the object detection result containing the local information of the object.

[0050] In the above steps S201-S207, steps S202-S207 combine Figure 1 The object detection method has already been described in detail, so it will not be repeated here.

[0051] In step S201, the neural network model consisting of the residual network, the feature pyramid network, and the classification and regression network is trained.

[0052] The operations performed by this neural network model can be referred to... Figure 2 Steps S203-S205. Also, refer to... Figure 3 , Figure 3 This is a schematic structural framework diagram illustrating a neural network model according to an embodiment of the present disclosure. Figure 3 As shown, in this neural network model, the residual network first extracts three shallow features at different scales related to the first and second objects from the image to be detected, and outputs these three shallow features to the feature pyramid network. Then, the feature pyramid network fuses the three shallow features to obtain the fused features of the first and second objects, and transmits all features of each layer of the first and second objects to the classification regression network (Classification Regression Network 1, Classification Regression Network 3, or Classification Regression Network 5) for the first object and the classification regression network (Classification Regression Network 2, Classification Regression Network 4, or Classification Regression Network 6) for the second object. Finally, the classification regression network for the first object outputs a prediction result of the first object with local information of the first object, combining the above information... Figure 2 As the description indicates, the prediction result for the first object corresponds to the preliminary detection result for the first object; the classification regression network for the second object outputs a prediction result for the second object containing local information about the second object. Combined with the above information... Figure 2 As explained, the prediction result for the second object corresponds to the preliminary detection result of the second object. Therefore, the output of this neural network model is combined with... Figure 2 Steps S206 and S207 in the process can obtain the final detection results of the first object and the second object.

[0053] It should be understood that the above combination Figure 3 The described neural network model is merely exemplary. The residual network and feature pyramid network may not have three feature layers; instead, they can be configured to output feature layers of other numbers as needed. Furthermore, the number of objects to be detected is not limited to two; it can be any number of other numbers. Therefore, the number of classification and regression networks can be adaptively adjusted based on the number of feature layers and the number of objects to be detected. According to embodiments of this disclosure, the number of classification and regression networks is equal to the product of the number of feature layers and the number of objects to be detected.

[0054] It should be understood that if you want to use a neural network model to implement the above method and achieve good accuracy, you need to train it.

[0055] Specifically, training a neural network model may include: when detecting a single object, training the neural network model using all labeled training data for that single object in a labeled training set; when detecting multiple objects, training the neural network model using all labeled training data for those multiple objects in a labeled training set, and training the neural network model separately using labeled training data for each object in the labeled training set.

[0056] Furthermore, taking two objects to be detected—a person and a vehicle—as an example, with the person's head as the local area and the vehicle's license plate as the local area, the training of the neural network model can include the following steps:

[0057] First, set the initial learning rate to α, which can be between 0.01 and 0.05. For example, α = 0.01. Use all labeled training data in the labeled training set for people and heads, as well as cars and license plates, to train the neural network model for m epochs. The value of m is, for example, 12. Of course, other values ​​can be used as needed.

[0058] Second, the learning rate is adjusted to 0.1α, the weights of the residual network, the feature pyramid network, and the classification and regression network for people and heads are fixed, and the neural network model is trained using labeled training data for cars and license plates in the labeled training set, and trained for m / 4 epochs, for example, 3 epochs when m=12.

[0059] Third, the learning rate is still set to 0.1α, the weights of the residual network, the feature pyramid network, and the classification and regression network for vehicles and license plates are fixed, and the neural network model is trained using labeled training data for people and heads in the labeled training set, and trained for m / 3 epochs, for example, 4 epochs when m=12.

[0060] Fourth, the above three steps can be repeated three times to complete the training of the neural network model.

[0061] In the training of the neural network model described above, the loss of the neural network model can be determined using classification loss functions and regression loss functions based on the object prediction results of the neural network model and the corresponding labeled training data. The classification loss function can include the Focal Loss function, and the regression loss function can include the IouLoss function. Furthermore, the loss can include the sum of the classification losses for all object prediction results and the sum of the coordinate losses for all positive object prediction results and the coordinate losses for all positive samples.

[0062] Specifically, supervised training is used for the neural network model, and the labels of the training set are set as follows: The position of the corresponding anchor (box) is set to (ct x ,ct y ,w,h), therefore the regression objective (the output of the neural network theory) It can be calculated using the following formula:

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071] Where x0, y0, x1, y1 are the coordinate parameters of the original bounding box (label box) of the object, and c is the classification parameter of the object. c represents the coordinate parameters of the original bounding box (annotation box) for a local part of the object. e For local classification parameters of the object; ct x ,ct y ,w,h represent the preset anchor's center coordinates and dimensions; t x ,t y ,t w ,t h This represents the coordinate parameters in the object detection results that the neural network model should theoretically output. This represents the coordinate parameters in the local object detection results that the neural network model should theoretically output; constant x constant y ,scale w ,scale h These are constants derived from experiments and empirical statistics.

[0072] Furthermore, the prediction result of the neural network model, i.e., the actual output result, can be set as... Where, r x ,r y ,r w ,r hThe coordinate parameters represent the actual object predictions made by the neural network model, where p is the object's classification parameter. p represents the coordinate parameters in the local result of the object actually predicted by the neural network model. e The classification parameters for the object.

[0073] Therefore, for the object in question, the loss can be calculated using the following formula by comparing the actual output P and the theoretical output T of the neural network:

[0074]

[0075] Where Focalloss and Iouloss are the loss functions Focal Loss and Iou Loss, respectively; α represents the learning rate as described above; anchors represent the preset number of anchors for the objects involved; pos represents all positive samples in the labeled training set of the objects involved; and r represents the learning rate. x ,r y ,r w ,r h t represents t x ,t y ,t w ,t h r e represent t e represent {c e >0}Iouloss(r e ,t e ) represents only c e The loss Iouloss(r) is calculated only when the value is greater than 0 (i.e., when the local sample of the object involved is positive). e ,t e ).

[0076] Therefore, the loss may include the sum of the classification loss of the object and the object locality in the object prediction results for all anchors (all samples) of the involved object, and the sum of the coordinate loss of the object and the object locality in the object prediction results for all positive samples of the involved object and the object locality.

[0077] This disclosure also provides an object detection apparatus. This object detection apparatus is used to perform the steps in the above-described object detection method embodiments.

[0078] Reference Figure 4 , Figure 4This is a schematic block diagram illustrating an object detection apparatus 100 according to an embodiment of the present disclosure. The object detection apparatus 100 includes an acquisition module 101, a feature extraction module 102, and a detection module 103. The acquisition module 101 is configured to acquire an image to be detected. The feature extraction module 102 is configured to obtain object features associated with at least one object based on the image to be detected. The detection module 103 is configured to determine an object detection result with local object information for the at least one object based on the object features.

[0079] According to an embodiment of this disclosure, the feature extraction module 102 can be configured to obtain object features related to the at least one object from the image to be detected in the following manner: extracting shallow features related to the at least one object from the image to be detected through a backbone network; and fusing the shallow features through a feature fusion network to obtain the object features.

[0080] Furthermore, the feature extraction module 102 can also be configured to obtain object features related to the at least one object from the image to be detected in the following manner: extracting multi-layer shallow features of different scales related to the at least one object from the image to be detected through a backbone network; and fusing the multi-layer shallow features respectively through a feature pyramid network to obtain multi-layer object features.

[0081] According to embodiments of this disclosure, the backbone network may be a residual network.

[0082] According to an embodiment of this disclosure, the detection module 103 can be configured to determine an object detection result with local object information for each of the at least one object based on the object features in the following manner: obtaining a preliminary object detection result with local object information for each of the at least one object through a respective classification and regression network based on the object features; obtaining an intermediate object detection result by non-maximum suppression processing based on the preliminary object detection result; and determining an object detection result with local object information based on the intermediate object detection result.

[0083] Furthermore, the detection module 103 can be configured to determine an object detection result containing the local information of the object based on the intermediate object detection result in the following manner: when the multiplication of the classification parameter of the object in the intermediate object detection result and the classification parameter of the local object is greater than a preset threshold, the intermediate object detection result is taken as the object detection result containing the local information of the object.

[0084] This disclosure also provides another object detection apparatus. This object detection apparatus is used to perform the steps in the above-described object detection method embodiments.

[0085] Reference Figure 5 , Figure 5 This is a schematic block diagram illustrating an object detection apparatus 200 according to another embodiment of the present disclosure. Figure 5 The object detection device 200 shown and Figure 4 The object detection device 100 shown differs only in that the object detection device 200 also includes a training module 201. This training module 201 is configured to train the neural network model composed of the aforementioned residual network, feature pyramid network, and classification regression network.

[0086] According to an embodiment of this disclosure, the training module 201 is configured to train the neural network model in the following manner: when detecting a single object, the neural network model is trained using all labeled training data for that single object in the labeled training set; when detecting multiple objects, the neural network model is trained using all labeled training data for the multiple objects in the labeled training set, and the neural network model is trained separately using labeled training data for each object in the labeled training set.

[0087] According to embodiments of this disclosure, the loss of the neural network model is determined by using a classification loss function and a regression loss function based on the object prediction results of the neural network model and the corresponding labeled training data.

[0088] Furthermore, the loss includes the sum of the classification loss of objects and object localities in all object prediction results, and the sum of the coordinate loss of objects and object localities in all positive sample object prediction results.

[0089] Furthermore, the classification loss function includes the Focal Loss function, and the regression loss function includes the IouLoss function.

[0090] It is understood that the specific way in which each module performs its operation in the object detection device in the above embodiments has been described in detail in the embodiments of the object detection method, and will not be elaborated here.

[0091] Figure 6 This is a structural diagram illustrating a combined processing apparatus 600 according to an embodiment of the present disclosure. As shown, the combined processing apparatus 600 includes a computing device 602, which may include the object detection device of the present disclosure as described above in conjunction with the accompanying drawings. Additionally, the combined processing apparatus includes a general interconnect interface 604 and other processing devices 606. The computing device according to the present disclosure interacts with other processing devices to jointly complete user-specified operations.

[0092] According to the scheme of this disclosure, the other processing device may include one or more types of processors such as a central processing unit (“CPU”), a graphics processing unit (“GPU”), a neural network processor, and other general-purpose and / or special-purpose processors, and the number is not limited but determined according to actual needs. In one or more embodiments, the other processing device may serve as an interface between the computing device of this disclosure (which may be embodied as a machine learning computing device) and external data and control, performing tasks including but not limited to data transfer, and completing basic control such as starting and stopping the machine learning computing device; the other processing device may also cooperate with the machine learning computing device to jointly complete computing tasks.

[0093] According to the present disclosure, the universal interconnect interface can be used to transfer data and control commands between a computing device and other processing devices. For example, the computing device can obtain required input data from other processing devices via the universal interconnect interface and write it to on-chip storage. Furthermore, the computing device can obtain control commands from other processing devices via the universal interconnect interface and write them to on-chip control cache. Alternatively or optionally, the universal interconnect interface can also read data from the computing device's storage module and transmit it to other processing devices.

[0094] Optionally, the combined processing apparatus may further include a storage device 608, which may be connected to the computing device and the other processing device respectively. In one or more embodiments, the storage device may be used to store data from the computing device and the other processing device, particularly suitable for data that cannot be fully stored in the internal storage of the computing device or other processing device, requiring computation.

[0095] Depending on the application scenario, the combined processing device disclosed herein can serve as a System-on-a-Chip (SoC) for devices such as mobile phones, robots, drones, video capture equipment, and video surveillance equipment, thereby effectively saving computing power and offering strong applicability. In this case, the general interconnect interface of the combined processing device is connected to certain components of the device. These components may include, for example, a camera, monitor, mouse, keyboard, network card, or Wi-Fi interface.

[0096] In some embodiments, this disclosure also discloses a chip or integrated circuit chip that includes the aforementioned computing device, combined processing device, and object detection device. In other embodiments, this disclosure also discloses a chip package structure that includes the aforementioned chip.

[0097] In some embodiments, this disclosure also discloses a circuit board that includes the above-described chip packaging structure. See also... Figure 7The aforementioned exemplary board 700 is provided. In addition to the aforementioned chip 702, the board may also include other supporting components, which may include, but are not limited to, a storage device 704, an interface device 706, and a controller 708.

[0098] The storage device is connected to the chip within the chip package structure via a bus for storing data. The storage device may include multiple sets of storage cells 710. Each set of storage cells is connected to the chip via a bus. It is understood that each set of storage cells may be DDR SDRAM (“Double Data Rate SDRAM”).

[0099] DDR can double the speed of SDRAM without increasing the clock frequency. DDR allows data to be read on both the rising and falling edges of the clock pulse. DDR is twice as fast as standard SDRAM. In one embodiment, the memory device may include four groups of memory cells. Each group of memory cells may include multiple DDR4 chips. In one embodiment, the chip may internally include four 72-bit DDR4 controllers, of which 64 bits are used for data transmission and 8 bits are used for ECC verification.

[0100] In one embodiment, each group of memory cells may include multiple Double Data Rate (DDR) synchronous dynamic random access memories (DRAMs) arranged in parallel. DDR can transfer data twice within one clock cycle. A controller for controlling the DDR is provided in the chip for controlling the data transfer and data storage of each memory cell.

[0101] The interface device is electrically connected to the chip within the chip package structure. The interface device is used to realize data transmission between the chip and an external device 712 (e.g., a server or computer). For example, in one embodiment, the interface device can be a standard PCIe interface. For instance, data to be processed is transferred from the server to the chip via a standard PCIe interface, realizing data transfer. In another embodiment, the interface device can also be other interfaces; this disclosure does not limit the specific form of the other interfaces mentioned above, as long as the interface unit can realize the switching function. Furthermore, the calculation results of the chip are still transmitted back to the external device (e.g., the server) by the interface device.

[0102] The controller is electrically connected to the chip to monitor its status. Specifically, the chip and the controller can be electrically connected via an SPI interface. The controller may include a microcontroller ("MCU"). The chip may include multiple processing chips, multiple processing cores, or multiple processing circuits, and can drive multiple loads. Thus, the chip can operate in different states, such as high load and low load. The controller can regulate the operating states of multiple processing chips, multiple processing cores, and / or multiple processing circuits within the chip.

[0103] In some embodiments, this disclosure also discloses an electronic device or apparatus that includes the aforementioned circuit board. Depending on the application scenario, the electronic device or apparatus may include a data processing device, robot, computer, printer, scanner, tablet computer, smart terminal, mobile phone, dashcam, navigator, sensor, camera, server, cloud server, camera, camcorder, projector, watch, earphone, mobile storage, wearable device, vehicle, home appliance, and / or medical device. The vehicle includes airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs.

[0104] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0105] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0106] In the several embodiments provided in this disclosure, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the circuit division is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple circuits or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be indirect couplings or communication connections between devices or circuits through some interfaces, and may be electrical, optical, acoustic, magnetic, or other forms.

[0107] The circuit described as a separate component may or may not be physically separate. The components shown as circuits may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] Furthermore, the functional circuits in the various embodiments of this disclosure can be integrated into one processing unit, or each circuit can exist physically separately, or two or more circuits can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.

[0109] If the integrated unit is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, when the technical solution of this disclosure can be embodied in the form of a software product, the computer software product is stored in a storage device and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage device includes various media capable of storing program code, such as a USB flash drive, read-only memory ("ROM"), random access memory ("RAM"), portable hard drive, magnetic disk, or optical disk.

[0110] The foregoing can be better understood in accordance with the following terms:

[0111] Clause A1, an object detection method, wherein the object detection method includes: acquiring an image to be detected; obtaining object features associated with at least one object based on the image to be detected; and determining object detection results with local object information for each of the at least one object based on the object features.

[0112] Clause A2, the object detection method according to Clause A1, wherein obtaining object features related to the at least one object from the image to be detected includes: extracting shallow features related to the at least one object from the image to be detected through a backbone network; and fusing the shallow features through a feature fusion network to obtain the object features.

[0113] Clause A3, the object detection method according to Clause A2, wherein multi-layer shallow features of different scales related to the at least one object are extracted from the image to be detected by a backbone network; and the multi-layer shallow features are fused by a feature pyramid network to obtain multi-layer object features.

[0114] Clause A4, the object detection method according to Clause A3, wherein the backbone network is a residual network.

[0115] Clause A5, the object detection method according to Clause A4, wherein determining an object detection result with local object information for each of the at least one object based on the object features includes: obtaining a preliminary object detection result with local object information for each of the at least one object through a respective classification and regression network based on the object features; obtaining an intermediate object detection result by non-maximum suppression processing based on the preliminary object detection result; and determining an object detection result with local object information based on the intermediate object detection result.

[0116] Clause A6, according to the object detection method described in Clause A5, wherein determining the object detection result containing the object's local information based on the intermediate object detection result includes: when the product of the object's classification parameter and the object's local classification parameter in the intermediate object detection result is greater than a preset threshold, the intermediate object detection result is taken as the object detection result containing the object's local information.

[0117] Clause A7, the object detection method according to Clause A5, wherein the object detection method further includes: training a neural network model composed of the residual network, the feature pyramid network and the classification regression network.

[0118] Clause A8, the object detection method according to Clause A7, wherein the training includes: when detecting a single object, training the neural network model using all labeled training data for the single object in a labeled training set; when detecting multiple objects, training the neural network model using all labeled training data for the multiple objects in a labeled training set, and training the neural network model separately using labeled training data for each object in a labeled training set.

[0119] Clause A9, the object detection method according to Clause A8, wherein the loss of the neural network model is determined by a classification loss function and a regression loss function based on the object prediction results of the neural network model and the corresponding labeled training data.

[0120] Clause A10, the object detection method according to Clause A9, wherein the loss includes the sum of the classification loss of objects and object localities in all object prediction results and the sum of the coordinate loss of objects and object localities in all positive sample object prediction results.

[0121] Clause A11, the object detection method according to Clause A10, wherein the classification loss function includes the FocalLoss function and the regression loss function includes the Iou Loss function.

[0122] Clause A12, an object detection apparatus, wherein the object detection apparatus comprises: an acquisition module for acquiring an image to be detected; a feature extraction module for obtaining object features related to at least one object based on the image to be detected; and a detection module for determining an object detection result with local object information for the at least one object based on the object features.

[0123] Clause A13, an integrated circuit chip, including the object detection device as described in Clause A12.

[0124] Clause A14, a computing device, including the object detection device according to Clause A12 or the integrated circuit chip according to Clause A13.

[0125] Clause A15, a board including a computing device as described in Clause A14.

[0126] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

[0127] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0128] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0129] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0130] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this disclosure, and on the specific implementation methods and application scope of this disclosure, are all within the scope of protection of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A method of object detection, wherein, The object detection method comprises: acquiring a to-be-detected image; obtaining object features related to at least one object according to the to-be-detected image; determining, according to the object features, an object detection result with object local information for each of the at least one object, which specifically comprises: obtaining, according to the object features, a preliminary object detection result with the object local information for each of the at least one object through a respective classification regression network; and obtaining an intermediate object detection result through non-maximum suppression processing according to the preliminary object detection result, and taking the intermediate object detection result as the object detection result with the object local information when a product of a classification parameter of the object in the intermediate object detection result and a classification parameter of the object local part is greater than a preset threshold.

2. The object detection method according to claim 1, wherein The object features related to the at least one object are obtained according to the to-be-detected image, which comprises: extracting, through a backbone network, shallow features related to the at least one object from the to-be-detected image; fusing the shallow features through a feature fusion network to obtain the object features.

3. The object detection method according to claim 2, wherein different-scale multi-layer shallow features related to the at least one object are extracted from the to-be-detected image through a backbone network; the multi-layer shallow features are fused respectively through a feature pyramid network to obtain multi-layer object features.

4. The object detection method according to claim 3, wherein The backbone network is a residual network.

5. The object detection method of claim 4, wherein, The object detection method further comprises: training a neural network model composed of the residual network, the feature pyramid network and the classification regression network.

6. The object detection method of claim 5, wherein, The training comprises: when detecting one object, training the neural network model using all labeled training data for the one object in a labeled training set; when detecting multiple objects, training the neural network model using all labeled training data for the multiple objects in a labeled training set, and respectively training the neural network model using labeled training data for each object in a labeled training set.

7. The object detection method of claim 6, wherein, According to object prediction results of the neural network model and corresponding labeled training data, a classification loss function and a regression loss function are used to determine a loss of the neural network model.

8. The object detection method of claim 7, wherein, The loss comprises a sum of classification losses of objects and object local parts in all object prediction results and a sum of coordinate losses of objects and object local parts in object prediction results for all positive samples.

9. The object detection method of claim 8, wherein, The classification loss function comprises a Focal Loss function, and the regression loss function comprises an Iou Loss function.

10. An object detection apparatus, wherein, The object detection apparatus comprises: an acquisition module configured to acquire a to-be-detected image; a feature extraction module configured to obtain object features related to at least one object according to the to-be-detected image; The detection module is configured to: obtain, according to the object features, a preliminary object detection result with the object local information for the at least one object through a respective classification regression network; and obtain, according to the preliminary object detection result, an intermediate object detection result through a non-maximum suppression process, and when a product of a classification parameter of the object and a classification parameter of the object local part in the intermediate object detection result is greater than a preset threshold, take the intermediate object detection result as the object detection result with the object local information, wherein the classification parameter of the object includes a parameter indicating whether the object is the object, and the classification parameter of the object local part includes a parameter indicating whether the object local part is the object local part.

11. An integrated circuit chip comprising the object detection apparatus of claim 10.

12. A computing apparatus comprising the object detection apparatus of claim 10 or the integrated circuit chip of claim 11.

13. A board card comprising the computing apparatus of claim 12.

Images